[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:11 dist-test-slave-4pw5 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:11 dist-test-slave-4pw5 krb5kdc[399](info): set up 2 sockets
Sep 26 11:35:11 dist-test-slave-4pw5 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 11:35:12.806371 392 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.006s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/test-user.keytab.
W20260926 11:35:13.075110 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.163s user 0.001s sys 0.000s
Sep 26 11:35:13 dist-test-slave-4pw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:13.128732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43283
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:13.256853 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:13.257234 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:13.257318 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:13.263401 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:13.263535 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:13.263598 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:13.263631 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:13.263690 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:13.271955 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:13.273162 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:13.274295 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:13.284588 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:13.284591 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:13.285048 412 server_base.cc:1034] running on GCE node
W20260926 11:35:13.285180 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:13.285907 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:13.287276 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:13.288470 412 hybrid_clock.cc:648] HybridClock initialized: now 1790422513288446 us; error 45 us; skew 500 ppm
Sep 26 11:35:13 dist-test-slave-4pw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:13.291780 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:13.292856 412 webserver.cc:460] Webserver started at http://127.0.98.62:45923/ using document root <none> and password file <none>
I20260926 11:35:13.293491 412 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:13.293584 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:13.293839 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:13.296221 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "9d4f4204413c494d80134b90e4c10eb3"
format_stamp: "Formatted at 2026-09-26 11:35:13 on dist-test-slave-4pw5"
server_key: "22dc9cff7b19e00b38869b543159785f"
server_key_iv: "8520ab8ee0c3e8b02119fe97864979e8"
server_key_version: "encryptionkey@0"
I20260926 11:35:13.296820 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d4f4204413c494d80134b90e4c10eb3"
format_stamp: "Formatted at 2026-09-26 11:35:13 on dist-test-slave-4pw5"
server_key: "22dc9cff7b19e00b38869b543159785f"
server_key_iv: "8520ab8ee0c3e8b02119fe97864979e8"
server_key_version: "encryptionkey@0"
I20260926 11:35:13.301540 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:35:13.304499 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:13.305882 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:13.306037 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "9d4f4204413c494d80134b90e4c10eb3"
format_stamp: "Formatted at 2026-09-26 11:35:13 on dist-test-slave-4pw5"
server_key: "22dc9cff7b19e00b38869b543159785f"
server_key_iv: "8520ab8ee0c3e8b02119fe97864979e8"
server_key_version: "encryptionkey@0"
I20260926 11:35:13.306180 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:13.328796 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:13.333181 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:13.333442 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:13.342095 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43283
I20260926 11:35:13.342113 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43283 every 8 connection(s)
I20260926 11:35:13.343538 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:13.346808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 412
I20260926 11:35:13.347092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:13.347394 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:13.348587 392 external_mini_cluster.cc:1428] Setting key 08f6b6d55133ca2112acb17e1b735275
I20260926 11:35:13.354611 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3: Bootstrap starting.
I20260926 11:35:13.357530 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:13.358417 480 log.cc:824] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:13.360756 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3: No bootstrap required, opened a new log
I20260926 11:35:13.364595 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4f4204413c494d80134b90e4c10eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260926 11:35:13.364887 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:13.364977 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d4f4204413c494d80134b90e4c10eb3, State: Initialized, Role: FOLLOWER
I20260926 11:35:13.365471 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4f4204413c494d80134b90e4c10eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260926 11:35:13.365640 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:13.365734 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:13.365865 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:13.366972 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4f4204413c494d80134b90e4c10eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260926 11:35:13.367367 480 leader_election.cc:302] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d4f4204413c494d80134b90e4c10eb3; no voters:
I20260926 11:35:13.367798 480 leader_election.cc:288] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:13.367965 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:13.368238 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 9d4f4204413c494d80134b90e4c10eb3, State: Running, Role: LEADER
Sep 26 11:35:13 dist-test-slave-4pw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:13.368588 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4f4204413c494d80134b90e4c10eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260926 11:35:13.369304 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:13.371506 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d4f4204413c494d80134b90e4c10eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4f4204413c494d80134b90e4c10eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } } }
I20260926 11:35:13.371688 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:13.372057 495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:13.371999 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d4f4204413c494d80134b90e4c10eb3. Latest consensus state: current_term: 1 leader_uuid: "9d4f4204413c494d80134b90e4c10eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4f4204413c494d80134b90e4c10eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } } }
I20260926 11:35:13.372197 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:13.375824 495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:13.377812 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:13.352170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.98.62:43283)
0926 11:35:13.352677 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:13.352695 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:13.353861 (+ 1166us) server_negotiation.cc:407] Connection header received
0926 11:35:13.358211 (+ 4350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:13.358231 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:13.358676 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:13.359038 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:13.361563 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.362590 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.363961 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.364450 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.364647 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:13.369736 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:13.369748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:13.369767 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:13.369809 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:13.372464 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:13.373323 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:13.373329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:13.373334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:13.373472 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:13.373943 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:13.373947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:13.373948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:13.374377 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:13.374633 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:13.376774 (+ 2141us) server_negotiation.cc:299] Negotiation successful
0926 11:35:13.377046 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":162,"threads_started":1}
I20260926 11:35:13.383754 495 catalog_manager.cc:1347] Generated new cluster ID: 6fe768aa25b94e80b26e28f5940062a9
I20260926 11:35:13.383857 495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:13.402585 495 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:13.404170 495 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:13.415800 495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d4f4204413c494d80134b90e4c10eb3: Generated new TSK 0
I20260926 11:35:13.416708 495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:13.455534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:13.586798 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:13.587155 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:13.587208 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:13.591611 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:13.591694 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:13.591760 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:13.597491 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:13.598997 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:13.600386 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:13.612509 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:13.612619 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:13.612685 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:13.612871 506 server_base.cc:1034] running on GCE node
I20260926 11:35:13.613301 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:13.614030 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:13.615382 506 hybrid_clock.cc:648] HybridClock initialized: now 1790422513615378 us; error 49 us; skew 500 ppm
Sep 26 11:35:13 dist-test-slave-4pw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:13.618929 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:13.620054 506 webserver.cc:460] Webserver started at http://127.0.98.1:38201/ using document root <none> and password file <none>
I20260926 11:35:13.620677 506 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:13.620780 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:13.621017 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:13.623551 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "8749a44cd0304a37a2bcdcda2d384fba"
format_stamp: "Formatted at 2026-09-26 11:35:13 on dist-test-slave-4pw5"
server_key: "1097007fe7eecd67a8bd616184d64046"
server_key_iv: "a11907d6e3ffb2e94480c03c818705e4"
server_key_version: "encryptionkey@0"
I20260926 11:35:13.624315 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8749a44cd0304a37a2bcdcda2d384fba"
format_stamp: "Formatted at 2026-09-26 11:35:13 on dist-test-slave-4pw5"
server_key: "1097007fe7eecd67a8bd616184d64046"
server_key_iv: "a11907d6e3ffb2e94480c03c818705e4"
server_key_version: "encryptionkey@0"
I20260926 11:35:13.628541 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:13.631776 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:13.633407 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:35:13.633589 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "8749a44cd0304a37a2bcdcda2d384fba"
format_stamp: "Formatted at 2026-09-26 11:35:13 on dist-test-slave-4pw5"
server_key: "1097007fe7eecd67a8bd616184d64046"
server_key_iv: "a11907d6e3ffb2e94480c03c818705e4"
server_key_version: "encryptionkey@0"
I20260926 11:35:13.633744 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:13.648836 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:13.652797 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:13.653007 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:13.653717 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:13.654577 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:13.654634 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:13.654685 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:13.654701 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:13.664361 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43677
I20260926 11:35:13.664376 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43677 every 8 connection(s)
I20260926 11:35:13.665314 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:13.674413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 506
I20260926 11:35:13.674552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:13.674918 392 external_mini_cluster.cc:1428] Setting key 3abd2a55cdc4e74d82974b4baefc6a6c
Sep 26 11:35:13 dist-test-slave-4pw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:13.683028 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:13.667206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60587 (local address 127.0.98.62:43283)
0926 11:35:13.667349 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:13.667353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:13.668270 (+ 917us) server_negotiation.cc:407] Connection header received
0926 11:35:13.669707 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:13.669710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:13.669761 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:13.669840 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:13.671260 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.672017 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.673336 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.673584 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.673712 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:13.677452 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:13.677456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:13.677458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:13.677492 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:13.679771 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:13.680490 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:13.680498 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:13.680500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:13.680612 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:13.681087 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:13.681091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:13.681093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:13.681315 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:13.681403 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:13.682405 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0926 11:35:13.682736 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:35:13.684090 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:13.667556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43283 (local address 127.0.98.1:60587)
0926 11:35:13.668098 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:13.668154 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:13.669486 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:13.670017 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:13.670024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:13.670522 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:13.671053 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:13.671067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.672169 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.672173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:13.673206 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:13.673217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.673784 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.673787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:13.673998 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:13.674974 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:13.675000 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:13.677207 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:13.679953 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:13.679959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:13.679975 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:13.680312 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:13.680730 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:13.680735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:13.680741 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:13.680977 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:13.681435 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:13.681446 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:13.681872 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 11:35:13.682280 (+ 408us) client_negotiation.cc:240] Negotiation successful
0926 11:35:13.682746 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260926 11:35:13.685658 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43283
I20260926 11:35:13.685956 635 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:13.686575 635 heartbeater.cc:511] Master 127.0.98.62:43283 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:35:13.688365 444 ts_manager.cc:194] Registered new tserver with Master: 8749a44cd0304a37a2bcdcda2d384fba (127.0.98.1:43677)
I20260926 11:35:13.689785 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:35:13.708222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:13.835170 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:13.835490 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:13.835542 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:13.840085 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:13.840174 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:13.840245 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:13.846040 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:13.847201 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:13.848269 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:13.859985 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:13.860085 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:13.859997 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:13.860248 640 server_base.cc:1034] running on GCE node
I20260926 11:35:13.860770 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:13.861478 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:13.862677 640 hybrid_clock.cc:648] HybridClock initialized: now 1790422513862658 us; error 31 us; skew 500 ppm
Sep 26 11:35:13 dist-test-slave-4pw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:13.866111 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:35:13.867125 640 webserver.cc:460] Webserver started at http://127.0.98.2:36835/ using document root <none> and password file <none>
I20260926 11:35:13.867812 640 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:13.867909 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:13.868161 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:13.870568 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "8c79aea3380c423b8433a17cb8ee9736"
format_stamp: "Formatted at 2026-09-26 11:35:13 on dist-test-slave-4pw5"
server_key: "7c58ccc3be861df0d544487f12607654"
server_key_iv: "9ddc14a9858fe23c0015b44be1297231"
server_key_version: "encryptionkey@0"
I20260926 11:35:13.871342 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8c79aea3380c423b8433a17cb8ee9736"
format_stamp: "Formatted at 2026-09-26 11:35:13 on dist-test-slave-4pw5"
server_key: "7c58ccc3be861df0d544487f12607654"
server_key_iv: "9ddc14a9858fe23c0015b44be1297231"
server_key_version: "encryptionkey@0"
I20260926 11:35:13.875811 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:13.878720 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:13.880389 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:13.880561 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "8c79aea3380c423b8433a17cb8ee9736"
format_stamp: "Formatted at 2026-09-26 11:35:13 on dist-test-slave-4pw5"
server_key: "7c58ccc3be861df0d544487f12607654"
server_key_iv: "9ddc14a9858fe23c0015b44be1297231"
server_key_version: "encryptionkey@0"
I20260926 11:35:13.880707 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:13.903353 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:13.908133 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:13.908399 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:13.909101 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:13.910095 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:13.910210 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:13.910290 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:13.910336 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:13.921821 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34705
I20260926 11:35:13.921840 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34705 every 8 connection(s)
I20260926 11:35:13.922864 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:13.926492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 640
I20260926 11:35:13.926635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:13.926898 392 external_mini_cluster.cc:1428] Setting key 5672e6e994ac37daff6e6255384a5c7e
Sep 26 11:35:13 dist-test-slave-4pw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:35:13.941602 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:13.924928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60435 (local address 127.0.98.62:43283)
0926 11:35:13.925075 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:13.925084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:13.925956 (+ 872us) server_negotiation.cc:407] Connection header received
0926 11:35:13.927872 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:13.927874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:13.927939 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:13.928033 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:13.929697 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.930819 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.932645 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.932910 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.933038 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:13.936532 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:13.936540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:13.936546 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:13.936593 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:13.938841 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:13.939563 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:13.939608 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:13.939614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:13.939748 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:13.940259 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:13.940267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:13.940271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:13.940563 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:13.940665 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:13.941300 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 11:35:13.941458 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:35:13.942715 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:13.925278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43283 (local address 127.0.98.2:60435)
0926 11:35:13.925784 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:13.925831 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:13.927566 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:13.928262 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:13.928272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:13.928896 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:13.929505 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:13.929522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.931000 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.931006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:13.932491 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:13.932504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:13.933110 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:13.933118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:13.933393 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:13.934216 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:13.934248 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:13.936269 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:13.939038 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:13.939046 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:13.939070 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:13.939407 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:13.939886 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:13.939888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:13.939890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:13.940112 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:13.940674 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:13.940680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:13.940974 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 11:35:13.941156 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 11:35:13.941569 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:35:13.944167 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43283
I20260926 11:35:13.944429 769 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:13.944993 769 heartbeater.cc:511] Master 127.0.98.62:43283 requested a full tablet report, sending...
I20260926 11:35:13.946640 444 ts_manager.cc:194] Registered new tserver with Master: 8c79aea3380c423b8433a17cb8ee9736 (127.0.98.2:34705)
I20260926 11:35:13.947328 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:35:13.963444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:14.096657 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:14.096973 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:14.097019 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:14.101372 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:14.101449 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:14.101512 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:14.107146 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:14.108457 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:14.109694 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:14.120121 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:14.120296 774 server_base.cc:1034] running on GCE node
W20260926 11:35:14.120167 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:14.120123 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:14.121009 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:14.121711 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:14.122911 774 hybrid_clock.cc:648] HybridClock initialized: now 1790422514122890 us; error 36 us; skew 500 ppm
Sep 26 11:35:14 dist-test-slave-4pw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:14.125998 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:35:14.126955 774 webserver.cc:460] Webserver started at http://127.0.98.3:35323/ using document root <none> and password file <none>
I20260926 11:35:14.127556 774 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:14.127712 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:14.127954 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:14.130096 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "2484f4e65d9d47c79054b17827bb2836"
format_stamp: "Formatted at 2026-09-26 11:35:14 on dist-test-slave-4pw5"
server_key: "f039b254c570d5a32f754d063e38f0c0"
server_key_iv: "b12924be6207e454e1ecddcd20e29db5"
server_key_version: "encryptionkey@0"
I20260926 11:35:14.130743 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2484f4e65d9d47c79054b17827bb2836"
format_stamp: "Formatted at 2026-09-26 11:35:14 on dist-test-slave-4pw5"
server_key: "f039b254c570d5a32f754d063e38f0c0"
server_key_iv: "b12924be6207e454e1ecddcd20e29db5"
server_key_version: "encryptionkey@0"
I20260926 11:35:14.134989 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 11:35:14.137976 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:14.139515 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:14.139701 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "2484f4e65d9d47c79054b17827bb2836"
format_stamp: "Formatted at 2026-09-26 11:35:14 on dist-test-slave-4pw5"
server_key: "f039b254c570d5a32f754d063e38f0c0"
server_key_iv: "b12924be6207e454e1ecddcd20e29db5"
server_key_version: "encryptionkey@0"
I20260926 11:35:14.139840 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:14.158023 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:14.161844 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:14.162205 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:14.162916 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:14.163949 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:14.164033 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:14.164108 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:14.164160 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:14.175261 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35557
I20260926 11:35:14.175276 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35557 every 8 connection(s)
I20260926 11:35:14.176278 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:14.181136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 774
I20260926 11:35:14.181303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:14.181612 392 external_mini_cluster.cc:1428] Setting key da13987eef5aff89055f672c1412daea
Sep 26 11:35:14 dist-test-slave-4pw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:14.191664 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:14.178332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58641 (local address 127.0.98.62:43283)
0926 11:35:14.178505 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:14.178526 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:14.179344 (+ 818us) server_negotiation.cc:407] Connection header received
0926 11:35:14.180835 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:14.180838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:14.180896 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:14.180983 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:14.182468 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:14.183123 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:14.184706 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:14.184944 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:14.185077 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:14.187858 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:14.187861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:14.187863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:14.187889 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:14.189319 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:14.189919 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:14.189922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:14.189924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:14.190022 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:14.190449 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:14.190452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:14.190454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:14.190654 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:14.190766 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:14.191343 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 11:35:14.191464 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 11:35:14.192601 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:14.178697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43283 (local address 127.0.98.3:58641)
0926 11:35:14.179184 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:14.179227 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:14.180601 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:14.181185 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:14.181193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:14.181659 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:14.182306 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:14.182330 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:14.183250 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:14.183254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:14.184487 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:14.184496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:14.185131 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:14.185135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:14.185309 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:14.186017 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:14.186035 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:14.187681 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:14.189453 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:14.189458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:14.189475 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:14.189767 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:14.190160 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:14.190163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:14.190164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:14.190309 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:14.190767 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:14.190773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:14.191081 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 11:35:14.191258 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 11:35:14.191537 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260926 11:35:14.194007 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43283
I20260926 11:35:14.194260 903 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:14.194880 903 heartbeater.cc:511] Master 127.0.98.62:43283 requested a full tablet report, sending...
I20260926 11:35:14.196471 444 ts_manager.cc:194] Registered new tserver with Master: 2484f4e65d9d47c79054b17827bb2836 (127.0.98.3:35557)
I20260926 11:35:14.197103 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58641
I20260926 11:35:14.209231 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 11:35:14 dist-test-slave-4pw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422514, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:35:14 dist-test-slave-4pw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422514, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:35:14.238497 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:14.227796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.0.98.1:43677)
0926 11:35:14.228163 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:14.228171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:14.228200 (+ 29us) server_negotiation.cc:407] Connection header received
0926 11:35:14.228281 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:14.228286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:14.228483 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:14.228615 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:14.229295 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:14.230094 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:14.231071 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:14.231474 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:14.231660 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:14.234426 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:14.234436 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:14.234451 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:14.234482 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:14.236360 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:14.236866 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:14.236872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:14.236883 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:14.237042 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:14.237382 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:14.237385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:14.237387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:14.237704 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:14.237882 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:14.238180 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 11:35:14.238323 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":168,"threads_started":1}
W20260926 11:35:14.239723 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37882
Sep 26 11:35:14 dist-test-slave-4pw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 11:35:14 dist-test-slave-4pw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:35:14.268847 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:14.258771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.0.98.1:43677)
0926 11:35:14.258949 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:14.258955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:14.259117 (+ 162us) server_negotiation.cc:407] Connection header received
0926 11:35:14.259321 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:14.259324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:14.259374 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:14.259469 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:14.260311 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:14.260840 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:14.261730 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:14.261958 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:14.262082 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:14.264894 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:14.264898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:14.264900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:14.264930 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:14.266874 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:14.267418 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:14.267421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:14.267424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:14.267547 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:14.267896 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:14.267900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:14.267902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:14.268102 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:14.268278 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:14.268517 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 11:35:14.268670 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 11:35:14.270021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 506
I20260926 11:35:14.276417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 640
I20260926 11:35:14.285339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 774
I20260926 11:35:14.291306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 412
2026-09-26T11:35:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2617 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:14 dist-test-slave-4pw5 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:14 dist-test-slave-4pw5 krb5kdc[917](info): set up 2 sockets
Sep 26 11:35:14 dist-test-slave-4pw5 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 11:35:15.367926 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:15 dist-test-slave-4pw5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:15.515992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36371
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:15.654973 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:15.655462 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:15.655560 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:15.660851 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:15.660992 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:15.661027 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:15.661069 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:15.661105 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:15.668041 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:15.669662 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:15.671221 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:15.682356 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:15.682380 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:15.682476 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:15.683007 930 server_base.cc:1034] running on GCE node
I20260926 11:35:15.683825 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:15.685355 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:15.686534 930 hybrid_clock.cc:648] HybridClock initialized: now 1790422515686507 us; error 46 us; skew 500 ppm
Sep 26 11:35:15 dist-test-slave-4pw5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:15.694340 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:15.695839 930 webserver.cc:460] Webserver started at http://127.0.98.62:37483/ using document root <none> and password file <none>
I20260926 11:35:15.696746 930 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:15.696880 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:15.697199 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:15.700344 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa"
format_stamp: "Formatted at 2026-09-26 11:35:15 on dist-test-slave-4pw5"
server_key: "2feea2a4db1a7087b32ea586d25f7d6f"
server_key_iv: "1dd4c29361990f2546c8b1c8d39b661c"
server_key_version: "encryptionkey@0"
I20260926 11:35:15.701160 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa"
format_stamp: "Formatted at 2026-09-26 11:35:15 on dist-test-slave-4pw5"
server_key: "2feea2a4db1a7087b32ea586d25f7d6f"
server_key_iv: "1dd4c29361990f2546c8b1c8d39b661c"
server_key_version: "encryptionkey@0"
I20260926 11:35:15.706704 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 11:35:15.710453 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:15.712412 930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:35:15.712622 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa"
format_stamp: "Formatted at 2026-09-26 11:35:15 on dist-test-slave-4pw5"
server_key: "2feea2a4db1a7087b32ea586d25f7d6f"
server_key_iv: "1dd4c29361990f2546c8b1c8d39b661c"
server_key_version: "encryptionkey@0"
I20260926 11:35:15.712785 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:15.732972 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:15.737462 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:15.737753 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:15.747071 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36371
I20260926 11:35:15.747114 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36371 every 8 connection(s)
I20260926 11:35:15.748355 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:15.752398 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:15.755534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 930
I20260926 11:35:15.755726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:15.756031 392 external_mini_cluster.cc:1428] Setting key 05c4888ef1305aad99048facf8755745
I20260926 11:35:15.761452 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa: Bootstrap starting.
I20260926 11:35:15.764390 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:15.765415 998 log.cc:824] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:35:15 dist-test-slave-4pw5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:15.768014 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa: No bootstrap required, opened a new log
I20260926 11:35:15.773102 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36371 } }
I20260926 11:35:15.773427 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:15.773491 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccc0d43797b4f1ba5a62ca56885d2fa, State: Initialized, Role: FOLLOWER
I20260926 11:35:15.773574 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:15.757698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.0.98.62:36371)
0926 11:35:15.758336 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:15.758360 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:15.758411 (+ 51us) server_negotiation.cc:407] Connection header received
0926 11:35:15.759184 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:15.759207 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:15.759822 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:15.760294 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:15.761582 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:15.762723 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:15.763742 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:15.764274 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:15.764487 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:15.767551 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:15.767558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:15.767605 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:15.767643 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:15.769889 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:15.770472 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:15.770480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:15.770487 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:15.770664 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:15.770992 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:15.770995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:15.770997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:15.771561 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:15.771814 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:15.772126 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 11:35:15.772479 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":146,"threads_started":1}
I20260926 11:35:15.774019 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36371 } }
I20260926 11:35:15.774312 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:15.774423 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:15.774534 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:15.775631 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36371 } }
I20260926 11:35:15.776067 998 leader_election.cc:302] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ccc0d43797b4f1ba5a62ca56885d2fa; no voters:
I20260926 11:35:15.776484 998 leader_election.cc:288] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:15.776636 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:15.776895 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [term 1 LEADER]: Becoming Leader. State: Replica: 2ccc0d43797b4f1ba5a62ca56885d2fa, State: Running, Role: LEADER
I20260926 11:35:15.777254 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36371 } }
I20260926 11:35:15.777909 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:15.779131 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ccc0d43797b4f1ba5a62ca56885d2fa. Latest consensus state: current_term: 1 leader_uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36371 } } }
I20260926 11:35:15.779284 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:15.779341 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc0d43797b4f1ba5a62ca56885d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36371 } } }
I20260926 11:35:15.779445 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:15.779981 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:15.783893 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:15.790922 1012 catalog_manager.cc:1347] Generated new cluster ID: f4a75339b5ec408ea7e6714cc360d546
I20260926 11:35:15.791054 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:15.808012 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:15.809525 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:15.822669 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ccc0d43797b4f1ba5a62ca56885d2fa: Generated new TSK 0
I20260926 11:35:15.823621 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:15.865254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36371
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:16.016884 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:16.017453 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:16.017623 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:16.022511 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:16.022734 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:16.022872 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:16.029038 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36371
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:16.030652 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:16.031986 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:16.043519 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:16.043502 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:16.043511 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:16.043735 1024 server_base.cc:1034] running on GCE node
I20260926 11:35:16.044673 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:16.045434 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:16.046665 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790422516046637 us; error 45 us; skew 500 ppm
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:16.050021 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:16.051211 1024 webserver.cc:460] Webserver started at http://127.0.98.1:44621/ using document root <none> and password file <none>
I20260926 11:35:16.051959 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:16.052042 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:16.052237 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:16.054859 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "906d9feb6bc6462d9d84b7844bf776c6"
format_stamp: "Formatted at 2026-09-26 11:35:16 on dist-test-slave-4pw5"
server_key: "a22a73e49a0f8b6cbb0af0b4c5ac0d71"
server_key_iv: "f2cbf82a6acd8ade9d3df627c4b0ffff"
server_key_version: "encryptionkey@0"
I20260926 11:35:16.055543 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "906d9feb6bc6462d9d84b7844bf776c6"
format_stamp: "Formatted at 2026-09-26 11:35:16 on dist-test-slave-4pw5"
server_key: "a22a73e49a0f8b6cbb0af0b4c5ac0d71"
server_key_iv: "f2cbf82a6acd8ade9d3df627c4b0ffff"
server_key_version: "encryptionkey@0"
I20260926 11:35:16.060328 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:35:16.063195 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:16.065045 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:16.065290 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "906d9feb6bc6462d9d84b7844bf776c6"
format_stamp: "Formatted at 2026-09-26 11:35:16 on dist-test-slave-4pw5"
server_key: "a22a73e49a0f8b6cbb0af0b4c5ac0d71"
server_key_iv: "f2cbf82a6acd8ade9d3df627c4b0ffff"
server_key_version: "encryptionkey@0"
I20260926 11:35:16.065439 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:16.082178 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:16.086833 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:16.087092 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:16.087898 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:16.088838 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:16.088887 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:16.088954 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:16.088987 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:16.099540 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40975
I20260926 11:35:16.099597 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40975 every 8 connection(s)
I20260926 11:35:16.100580 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:16.104611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 1024
I20260926 11:35:16.104745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:16.105074 392 external_mini_cluster.cc:1428] Setting key 880059ceb025a1469120da9eef86275b
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:35:16.117955 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.102860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37265 (local address 127.0.98.62:36371)
0926 11:35:16.103038 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.103044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.104042 (+ 998us) server_negotiation.cc:407] Connection header received
0926 11:35:16.105807 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.105811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.105871 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.105952 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:16.107770 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.108399 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.109894 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.110353 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.110482 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.113565 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:16.113570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:16.113572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:16.113602 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:16.115394 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.116093 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.116102 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.116103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.116186 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.116619 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.116622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.116623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.116832 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:16.116950 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.117600 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.117758 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 11:35:16.119078 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.103265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36371 (local address 127.0.98.1:37265)
0926 11:35:16.103879 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:16.103926 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:16.105578 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:16.106159 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:16.106167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:16.106608 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:16.107514 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.107543 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.108603 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.108609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.109742 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.109753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.110552 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.110556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.110752 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.111668 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:16.111689 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:16.113339 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:16.115543 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:16.115551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:16.115611 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:16.115959 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:16.116302 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:16.116305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:16.116307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:16.116472 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:16.117005 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:16.117013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:16.117320 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 11:35:16.117533 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 11:35:16.117918 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":117,"threads_started":1}
I20260926 11:35:16.120512 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36371
I20260926 11:35:16.120776 1153 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:16.121313 1153 heartbeater.cc:511] Master 127.0.98.62:36371 requested a full tablet report, sending...
I20260926 11:35:16.123137 962 ts_manager.cc:194] Registered new tserver with Master: 906d9feb6bc6462d9d84b7844bf776c6 (127.0.98.1:40975)
I20260926 11:35:16.124975 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:35:16.139825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36371
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:16.260783 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:16.261135 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:16.261185 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:16.265566 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:16.265671 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:16.265741 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:16.272200 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36371
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:16.273454 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:16.274605 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:35:16.286844 1158 server_base.cc:1034] running on GCE node
W20260926 11:35:16.286798 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:16.286767 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:16.286733 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:16.287693 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:16.288421 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:16.289651 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790422516289623 us; error 53 us; skew 500 ppm
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:16.292909 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:35:16.294030 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40947/ using document root <none> and password file <none>
I20260926 11:35:16.294890 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:16.295008 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:16.295295 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:16.297713 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "ed0cef8a7def45f482aeeb91c7907129"
format_stamp: "Formatted at 2026-09-26 11:35:16 on dist-test-slave-4pw5"
server_key: "e1cbfb68b33078455c5025934b017f29"
server_key_iv: "41de016f8c524ee0faca1508e8b659ea"
server_key_version: "encryptionkey@0"
I20260926 11:35:16.298401 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ed0cef8a7def45f482aeeb91c7907129"
format_stamp: "Formatted at 2026-09-26 11:35:16 on dist-test-slave-4pw5"
server_key: "e1cbfb68b33078455c5025934b017f29"
server_key_iv: "41de016f8c524ee0faca1508e8b659ea"
server_key_version: "encryptionkey@0"
I20260926 11:35:16.303150 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:35:16.306411 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:16.308122 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:16.308293 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "ed0cef8a7def45f482aeeb91c7907129"
format_stamp: "Formatted at 2026-09-26 11:35:16 on dist-test-slave-4pw5"
server_key: "e1cbfb68b33078455c5025934b017f29"
server_key_iv: "41de016f8c524ee0faca1508e8b659ea"
server_key_version: "encryptionkey@0"
I20260926 11:35:16.308437 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:16.337049 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:16.340993 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:16.341252 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:16.341979 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:16.342933 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:16.343015 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:16.343086 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:16.343135 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:16.353840 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39559
I20260926 11:35:16.353910 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39559 every 8 connection(s)
I20260926 11:35:16.355422 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:16.357822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 1158
I20260926 11:35:16.357969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:16.358254 392 external_mini_cluster.cc:1428] Setting key cbe1d142991a526f767a0fb9612b5503
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:35:16.371382 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.357887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47713 (local address 127.0.98.62:36371)
0926 11:35:16.358057 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.358062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.359058 (+ 996us) server_negotiation.cc:407] Connection header received
0926 11:35:16.360620 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.360627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.360695 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.360796 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:16.362136 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.362702 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.364045 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.364306 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.364434 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.367159 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:16.367164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:16.367168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:16.367207 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:16.368832 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.369487 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.369491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.369492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.369572 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.370057 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.370061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.370063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.370279 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:16.370421 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.371071 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.371212 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:35:16.372363 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.358264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36371 (local address 127.0.98.2:47713)
0926 11:35:16.358871 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:16.358932 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:16.360376 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:16.360972 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:16.360979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:16.361425 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:16.361948 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.361962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.362853 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.362857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.363901 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.363912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.364478 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.364483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.364679 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.365359 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:16.365377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:16.366943 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:16.368973 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:16.368980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:16.368999 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:16.369336 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:16.369715 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:16.369718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:16.369720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:16.369955 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:16.370422 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:16.370432 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:16.370808 (+ 376us) client_negotiation.cc:769] Sending connection context
0926 11:35:16.371030 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 11:35:16.371410 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":198,"threads_started":1}
I20260926 11:35:16.373783 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36371
I20260926 11:35:16.374049 1287 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:16.374571 1287 heartbeater.cc:511] Master 127.0.98.62:36371 requested a full tablet report, sending...
I20260926 11:35:16.375835 962 ts_manager.cc:194] Registered new tserver with Master: ed0cef8a7def45f482aeeb91c7907129 (127.0.98.2:39559)
I20260926 11:35:16.376895 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:35:16.393291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36371
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:16.512732 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:16.513042 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:16.513091 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:16.517889 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:16.517966 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:16.518031 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:16.523803 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36371
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:16.524900 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:16.525911 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:16.536396 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:16.536396 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:16.536437 1292 server_base.cc:1034] running on GCE node
W20260926 11:35:16.536398 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:16.537149 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:16.537855 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:16.539062 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790422516539041 us; error 45 us; skew 500 ppm
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:16.542557 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:35:16.543664 1292 webserver.cc:460] Webserver started at http://127.0.98.3:41883/ using document root <none> and password file <none>
I20260926 11:35:16.544306 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:16.544371 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:16.544597 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:16.547220 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569"
format_stamp: "Formatted at 2026-09-26 11:35:16 on dist-test-slave-4pw5"
server_key: "520f55c5bc6d5ece098992aee121c73a"
server_key_iv: "e5468a81939a8f43f2c4cb5564bf3ca8"
server_key_version: "encryptionkey@0"
I20260926 11:35:16.547978 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569"
format_stamp: "Formatted at 2026-09-26 11:35:16 on dist-test-slave-4pw5"
server_key: "520f55c5bc6d5ece098992aee121c73a"
server_key_iv: "e5468a81939a8f43f2c4cb5564bf3ca8"
server_key_version: "encryptionkey@0"
I20260926 11:35:16.552428 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260926 11:35:16.555261 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:16.556851 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:35:16.557029 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569"
format_stamp: "Formatted at 2026-09-26 11:35:16 on dist-test-slave-4pw5"
server_key: "520f55c5bc6d5ece098992aee121c73a"
server_key_iv: "e5468a81939a8f43f2c4cb5564bf3ca8"
server_key_version: "encryptionkey@0"
I20260926 11:35:16.557236 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:16.578343 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:16.582124 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:16.582372 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:16.583019 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:16.584015 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:16.584086 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:16.584153 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:16.584203 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:16.593811 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34111
I20260926 11:35:16.593828 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34111 every 8 connection(s)
I20260926 11:35:16.594864 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:16.601336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 1292
I20260926 11:35:16.601485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:16.601850 392 external_mini_cluster.cc:1428] Setting key 78257fef964774e423a3b884cb0bed10
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:16.611718 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.597326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55867 (local address 127.0.98.62:36371)
0926 11:35:16.597474 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.597478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.598341 (+ 863us) server_negotiation.cc:407] Connection header received
0926 11:35:16.599951 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.599955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.600021 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.600164 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:16.601718 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.602707 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.604132 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.604411 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.604528 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.607353 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:16.607357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:16.607359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:16.607385 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:16.609041 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.609704 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.609707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.609708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.609797 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.610398 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.610401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.610402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.610662 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:16.610760 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.611372 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.611494 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:35:16.612910 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.597559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36371 (local address 127.0.98.3:55867)
0926 11:35:16.598127 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:16.598192 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:16.599703 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:16.600359 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:16.600367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:16.600776 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:16.601493 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.601518 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.602880 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.602884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.604000 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.604010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.604583 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.604587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.604783 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.605416 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:16.605433 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:16.607158 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:16.609235 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:16.609242 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:16.609267 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:16.609573 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:16.609917 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:16.609920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:16.609921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:16.610249 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:16.610793 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:16.610799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:16.611095 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 11:35:16.611301 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 11:35:16.611813 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":126,"threads_started":1}
I20260926 11:35:16.614490 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36371
I20260926 11:35:16.614792 1421 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:16.615378 1421 heartbeater.cc:511] Master 127.0.98.62:36371 requested a full tablet report, sending...
I20260926 11:35:16.616581 962 ts_manager.cc:194] Registered new tserver with Master: e3a8f9b9eb69486b8eab3ea9cdb37569 (127.0.98.3:34111)
I20260926 11:35:16.617322 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55867
I20260926 11:35:16.628824 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:35:16.642067 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.633283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.0.98.62:36371)
0926 11:35:16.633449 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.633453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.633490 (+ 37us) server_negotiation.cc:407] Connection header received
0926 11:35:16.633625 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.633627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.633675 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.633771 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:16.634804 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.635477 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.636562 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.636837 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.636935 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.638562 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:16.638567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:16.638569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:16.638599 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:16.640228 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.640715 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.640720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.640722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.640817 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.641109 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.641114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.641116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.641359 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:16.641510 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.641767 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.641888 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 11:35:16.648555 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:16.652442 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:16.671113 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.663518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39559 (local address 127.0.0.1:59094)
0926 11:35:16.664797 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:16.664859 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:16.665011 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:16.665799 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:16.665801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:16.665824 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:16.666011 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.666017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.667536 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.667539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.669034 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.669044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.670331 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.670337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.670476 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.670516 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 11:35:16.670695 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 11:35:16.670918 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":269,"spinlock_wait_cycles":1920,"thread_start_us":112,"threads_started":1}
I20260926 11:35:16.671288 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.664247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.0.98.1:40975)
0926 11:35:16.664552 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.664560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.664962 (+ 402us) server_negotiation.cc:407] Connection header received
0926 11:35:16.665224 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.665234 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.665453 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.665614 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:16.666122 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.667177 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.669480 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.670299 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.670435 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.670650 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.671019 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.671127 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":121,"threads_started":1}
I20260926 11:35:16.671322 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.663676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34111 (local address 127.0.0.1:49692)
0926 11:35:16.664797 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:16.664859 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:16.665008 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:16.665626 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:16.665632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:16.665655 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:16.665810 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.665817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.667703 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.667710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.669213 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.669227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.670741 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.670746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.670931 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.670960 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 11:35:16.671050 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 11:35:16.671140 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":89,"thread_start_us":128,"threads_started":1}
I20260926 11:35:16.671676 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.663717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.0.98.2:39559)
0926 11:35:16.664529 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.664542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.664955 (+ 413us) server_negotiation.cc:407] Connection header received
0926 11:35:16.665207 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.665213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.665438 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.665573 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:16.666160 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.667402 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.669172 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.670104 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.670330 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.670543 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.670743 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.670876 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":153,"threads_started":1}
I20260926 11:35:16.671955 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.663832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.0.98.3:34111)
0926 11:35:16.664528 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.664542 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.665150 (+ 608us) server_negotiation.cc:407] Connection header received
0926 11:35:16.665218 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.665222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.665439 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.665581 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:16.665949 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.667320 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.669367 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.670208 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.670428 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.670666 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.671631 (+ 965us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.671773 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":126,"threads_started":1}
I20260926 11:35:16.672731 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.663518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40975 (local address 127.0.0.1:50568)
0926 11:35:16.664812 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:16.664860 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:16.665037 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:16.665799 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:16.665802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:16.665830 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:16.665997 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.666002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.667359 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.667367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.669321 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.669333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.670741 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.670745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.670860 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.670886 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:35:16.671712 (+ 826us) client_negotiation.cc:240] Negotiation successful
0926 11:35:16.672627 (+ 915us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219,"mutex_wait_us":1278,"spinlock_wait_cycles":4480,"thread_start_us":144,"threads_started":1}
I20260926 11:35:16.675038 1087 tablet_service.cc:1467] Processing CreateTablet for tablet c483e6738cc54cf09745f09541aea5d8 (DEFAULT_TABLE table=test-table [id=a03a37e949b04758b4d0c1ac7156ac26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:16.675088 1355 tablet_service.cc:1467] Processing CreateTablet for tablet c483e6738cc54cf09745f09541aea5d8 (DEFAULT_TABLE table=test-table [id=a03a37e949b04758b4d0c1ac7156ac26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:16.675088 1221 tablet_service.cc:1467] Processing CreateTablet for tablet c483e6738cc54cf09745f09541aea5d8 (DEFAULT_TABLE table=test-table [id=a03a37e949b04758b4d0c1ac7156ac26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:16.676267 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c483e6738cc54cf09745f09541aea5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:16.676285 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c483e6738cc54cf09745f09541aea5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:16.676285 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c483e6738cc54cf09745f09541aea5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:16.683168 1438 tablet_bootstrap.cc:492] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6: Bootstrap starting.
I20260926 11:35:16.685129 1439 tablet_bootstrap.cc:492] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569: Bootstrap starting.
I20260926 11:35:16.685660 1438 tablet_bootstrap.cc:654] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:16.686420 1438 log.cc:824] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:16.687252 1440 tablet_bootstrap.cc:492] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129: Bootstrap starting.
I20260926 11:35:16.687356 1439 tablet_bootstrap.cc:654] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:16.688622 1439 log.cc:824] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:16.689116 1438 tablet_bootstrap.cc:492] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6: No bootstrap required, opened a new log
I20260926 11:35:16.689322 1438 ts_tablet_manager.cc:1397] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:35:16.689525 1440 tablet_bootstrap.cc:654] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:16.690671 1439 tablet_bootstrap.cc:492] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569: No bootstrap required, opened a new log
I20260926 11:35:16.690820 1440 log.cc:824] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:16.690941 1439 ts_tablet_manager.cc:1397] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 11:35:16.693471 1440 tablet_bootstrap.cc:492] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129: No bootstrap required, opened a new log
I20260926 11:35:16.693686 1440 ts_tablet_manager.cc:1397] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 11:35:16.693785 1438 raft_consensus.cc:348] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } }
I20260926 11:35:16.694080 1438 raft_consensus.cc:374] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:16.694133 1438 raft_consensus.cc:729] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906d9feb6bc6462d9d84b7844bf776c6, State: Initialized, Role: FOLLOWER
I20260926 11:35:16.694690 1438 consensus_queue.cc:260] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } }
I20260926 11:35:16.695252 1439 raft_consensus.cc:348] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } }
I20260926 11:35:16.695523 1439 raft_consensus.cc:374] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:16.695619 1439 raft_consensus.cc:729] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a8f9b9eb69486b8eab3ea9cdb37569, State: Initialized, Role: FOLLOWER
I20260926 11:35:16.695784 1438 ts_tablet_manager.cc:1428] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260926 11:35:16.696185 1153 heartbeater.cc:503] Master 127.0.98.62:36371 was elected leader, sending a full tablet report...
I20260926 11:35:16.696203 1439 consensus_queue.cc:260] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } }
I20260926 11:35:16.696908 1421 heartbeater.cc:503] Master 127.0.98.62:36371 was elected leader, sending a full tablet report...
I20260926 11:35:16.697309 1439 ts_tablet_manager.cc:1428] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260926 11:35:16.697511 1440 raft_consensus.cc:348] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } }
I20260926 11:35:16.697803 1440 raft_consensus.cc:374] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:16.697863 1440 raft_consensus.cc:729] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0cef8a7def45f482aeeb91c7907129, State: Initialized, Role: FOLLOWER
I20260926 11:35:16.698300 1440 consensus_queue.cc:260] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } }
I20260926 11:35:16.699061 1287 heartbeater.cc:503] Master 127.0.98.62:36371 was elected leader, sending a full tablet report...
I20260926 11:35:16.699376 1440 ts_tablet_manager.cc:1428] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:35:16.753962 1444 raft_consensus.cc:482] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:16.754294 1444 raft_consensus.cc:504] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } }
I20260926 11:35:16.755690 1444 leader_election.cc:288] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed0cef8a7def45f482aeeb91c7907129 (127.0.98.2:39559), e3a8f9b9eb69486b8eab3ea9cdb37569 (127.0.98.3:34111)
I20260926 11:35:16.761925 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.756366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48415 (local address 127.0.98.2:39559)
0926 11:35:16.756529 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.756534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.756554 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:35:16.756767 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.756774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.756845 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.756905 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:16.757561 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.758457 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.760499 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.761252 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.761395 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.761471 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.761682 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.761765 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 11:35:16.761955 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.755936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39559 (local address 127.0.98.1:48415)
0926 11:35:16.756452 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:16.756471 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:16.756604 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:16.757092 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:16.757099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:16.757146 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:16.757387 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.757399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.758625 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.758631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.760319 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.760333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.761405 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.761409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.761524 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.761551 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 11:35:16.761642 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 11:35:16.761789 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":28,"spinlock_wait_cycles":896,"thread_start_us":184,"threads_started":1}
I20260926 11:35:16.762796 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c483e6738cc54cf09745f09541aea5d8" candidate_uuid: "906d9feb6bc6462d9d84b7844bf776c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0cef8a7def45f482aeeb91c7907129" is_pre_election: true
I20260926 11:35:16.763023 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.756099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38949 (local address 127.0.98.3:34111)
0926 11:35:16.756255 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.756263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.756690 (+ 427us) server_negotiation.cc:407] Connection header received
0926 11:35:16.756876 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.756879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.756933 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.757028 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:16.757453 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.758248 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.760708 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.761664 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.761792 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.761891 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.762723 (+ 832us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.762839 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:35:16.763250 1241 raft_consensus.cc:2454] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906d9feb6bc6462d9d84b7844bf776c6 in term 0.
I20260926 11:35:16.763724 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.755959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34111 (local address 127.0.98.1:38949)
0926 11:35:16.756564 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:16.756580 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:16.756752 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:16.757039 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:16.757042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:16.757060 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:16.757329 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.757340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.758427 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.758434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.760517 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:16.760534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.761958 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.761966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:16.762520 (+ 554us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.762557 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 11:35:16.763431 (+ 874us) client_negotiation.cc:240] Negotiation successful
0926 11:35:16.763518 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":26,"spinlock_wait_cycles":16256,"thread_start_us":191,"threads_started":1}
I20260926 11:35:16.763965 1043 leader_election.cc:302] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906d9feb6bc6462d9d84b7844bf776c6, ed0cef8a7def45f482aeeb91c7907129; no voters:
I20260926 11:35:16.764329 1444 raft_consensus.cc:2781] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:16.764444 1444 raft_consensus.cc:482] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:16.764497 1444 raft_consensus.cc:3037] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:16.764911 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c483e6738cc54cf09745f09541aea5d8" candidate_uuid: "906d9feb6bc6462d9d84b7844bf776c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" is_pre_election: true
I20260926 11:35:16.765254 1375 raft_consensus.cc:2454] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906d9feb6bc6462d9d84b7844bf776c6 in term 0.
I20260926 11:35:16.765944 1444 raft_consensus.cc:504] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } }
I20260926 11:35:16.766500 1444 leader_election.cc:288] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [CANDIDATE]: Term 1 election: Requested vote from peers ed0cef8a7def45f482aeeb91c7907129 (127.0.98.2:39559), e3a8f9b9eb69486b8eab3ea9cdb37569 (127.0.98.3:34111)
I20260926 11:35:16.766891 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c483e6738cc54cf09745f09541aea5d8" candidate_uuid: "906d9feb6bc6462d9d84b7844bf776c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0cef8a7def45f482aeeb91c7907129"
I20260926 11:35:16.766911 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c483e6738cc54cf09745f09541aea5d8" candidate_uuid: "906d9feb6bc6462d9d84b7844bf776c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569"
I20260926 11:35:16.767038 1375 raft_consensus.cc:3037] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:16.767043 1241 raft_consensus.cc:3037] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:16.768237 1241 raft_consensus.cc:2454] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906d9feb6bc6462d9d84b7844bf776c6 in term 1.
I20260926 11:35:16.768507 1375 raft_consensus.cc:2454] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906d9feb6bc6462d9d84b7844bf776c6 in term 1.
I20260926 11:35:16.768595 1043 leader_election.cc:302] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906d9feb6bc6462d9d84b7844bf776c6, ed0cef8a7def45f482aeeb91c7907129; no voters:
I20260926 11:35:16.768805 1444 raft_consensus.cc:2781] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:16.769059 1444 raft_consensus.cc:686] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [term 1 LEADER]: Becoming Leader. State: Replica: 906d9feb6bc6462d9d84b7844bf776c6, State: Running, Role: LEADER
I20260926 11:35:16.769421 1444 consensus_queue.cc:237] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } }
I20260926 11:35:16.773147 962 catalog_manager.cc:5495] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 906d9feb6bc6462d9d84b7844bf776c6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "906d9feb6bc6462d9d84b7844bf776c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0cef8a7def45f482aeeb91c7907129" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "906d9feb6bc6462d9d84b7844bf776c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a8f9b9eb69486b8eab3ea9cdb37569" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34111 } health_report { overall_health: UNKNOWN } } }
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:35:16.801290 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.790219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.0.98.1:40975)
0926 11:35:16.790458 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.790465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.790639 (+ 174us) server_negotiation.cc:407] Connection header received
0926 11:35:16.790828 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.790835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.790906 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.791019 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:16.792262 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.792927 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.793879 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.794099 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.794186 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.797311 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:16.797326 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:16.797343 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:16.797371 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:16.799127 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.799694 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.799701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.799703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.799793 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.800241 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.800244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.800246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.800537 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:16.800759 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.800961 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.801073 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:35:16.831806 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.821575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.0.98.1:40975)
0926 11:35:16.821754 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.821758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.821781 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:35:16.821940 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.821943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.821990 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.822145 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:16.822989 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.823652 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.824545 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.824795 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.824883 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.827685 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:16.827690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:16.827692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:16.827723 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:16.829426 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.829927 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.829932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.829934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.830033 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.830548 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.830553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.830555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.830798 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:16.830994 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.831367 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.831517 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260926 11:35:16.845973 1422 tablet.cc:2369] T c483e6738cc54cf09745f09541aea5d8 P e3a8f9b9eb69486b8eab3ea9cdb37569: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:16.852231 1154 tablet.cc:2369] T c483e6738cc54cf09745f09541aea5d8 P 906d9feb6bc6462d9d84b7844bf776c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260926 11:35:16.856881 1288 tablet.cc:2369] T c483e6738cc54cf09745f09541aea5d8 P ed0cef8a7def45f482aeeb91c7907129: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:35:16.860684 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:16.849609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.0.98.1:40975)
0926 11:35:16.849820 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:16.849824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:16.849849 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:35:16.849998 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:16.850001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:16.850048 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:16.850351 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:16.851155 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.851811 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.852994 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:16.853276 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:16.853382 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:16.856999 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:16.857004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:16.857008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:16.857048 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:16.858649 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.859177 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.859181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.859183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.859289 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:16.859679 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:16.859682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:16.859683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:16.859893 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:16.860035 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:16.860386 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 11:35:16.860512 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260926 11:35:16.862365 1067 tablet_service.cc:2908] Rejecting scan request for tablet c483e6738cc54cf09745f09541aea5d8: Uninitialized: clean time has not yet been initialized
I20260926 11:35:16.864835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 1024
I20260926 11:35:16.872325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 1158
I20260926 11:35:16.878329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 1292
I20260926 11:35:16.885138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 930
2026-09-26T11:35:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2593 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[1463](info): set up 2 sockets
Sep 26 11:35:16 dist-test-slave-4pw5 krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260926 11:35:18.008024 392 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:18 dist-test-slave-4pw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260926 11:35:18.638211 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.521s user 0.003s sys 0.000s
I20260926 11:35:18.638825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:18.757890 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:18.758234 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:18.758285 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:18.762444 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:18.762537 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 11:35:18.762559 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:18.762581 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:18.762601 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:18.762626 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:18.768422 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36489
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:18.769582 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:18.770663 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:18.779919 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:18.779924 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:18.779953 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:18.780042 1476 server_base.cc:1034] running on GCE node
I20260926 11:35:18.780856 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:18.782135 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:18.783345 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790422518783330 us; error 33 us; skew 500 ppm
Sep 26 11:35:18 dist-test-slave-4pw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:18.786731 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:18.787904 1476 webserver.cc:460] Webserver started at http://127.0.98.62:41227/ using document root <none> and password file <none>
I20260926 11:35:18.788519 1476 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:18.788585 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:18.788825 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:18.791181 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "a1b02ccab9d8483a8585cb9e89cb9999"
format_stamp: "Formatted at 2026-09-26 11:35:18 on dist-test-slave-4pw5"
server_key: "2cfda7f271a5eb425ad720d4c30aaabc"
server_key_iv: "b8217f54cc366bca295db086547c750a"
server_key_version: "encryptionkey@0"
I20260926 11:35:18.791828 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1b02ccab9d8483a8585cb9e89cb9999"
format_stamp: "Formatted at 2026-09-26 11:35:18 on dist-test-slave-4pw5"
server_key: "2cfda7f271a5eb425ad720d4c30aaabc"
server_key_iv: "b8217f54cc366bca295db086547c750a"
server_key_version: "encryptionkey@0"
I20260926 11:35:18.796036 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:18.798997 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:18.800516 1476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:18.800686 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "a1b02ccab9d8483a8585cb9e89cb9999"
format_stamp: "Formatted at 2026-09-26 11:35:18 on dist-test-slave-4pw5"
server_key: "2cfda7f271a5eb425ad720d4c30aaabc"
server_key_iv: "b8217f54cc366bca295db086547c750a"
server_key_version: "encryptionkey@0"
I20260926 11:35:18.800828 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:18.816411 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:18.820117 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:18.820379 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:18.829965 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36489
I20260926 11:35:18.829982 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36489 every 8 connection(s)
I20260926 11:35:18.831094 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:18.834322 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:18.836216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 1476
I20260926 11:35:18.836308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:18.836580 392 external_mini_cluster.cc:1428] Setting key 06d78dd85b8fc16870fd0afee9208096
I20260926 11:35:18.841598 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999: Bootstrap starting.
I20260926 11:35:18.844933 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:18.845948 1554 log.cc:824] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:35:18 dist-test-slave-4pw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:18.848615 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999: No bootstrap required, opened a new log
I20260926 11:35:18.853940 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b02ccab9d8483a8585cb9e89cb9999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260926 11:35:18.854290 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:18.854348 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b02ccab9d8483a8585cb9e89cb9999, State: Initialized, Role: FOLLOWER
I20260926 11:35:18.854758 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:18.838208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.0.98.62:36489)
0926 11:35:18.838659 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:18.838679 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:18.838724 (+ 45us) server_negotiation.cc:407] Connection header received
0926 11:35:18.839592 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:18.839645 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:18.840092 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:18.840504 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:18.841233 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:18.842854 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:18.843984 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:18.844784 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:18.844981 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:18.847898 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:18.847906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:18.847924 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:18.847960 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:18.850256 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:18.851315 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:18.851323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:18.851331 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:18.851514 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:18.851950 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:18.851954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:18.851956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:18.852599 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:18.852844 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:18.853251 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 11:35:18.853616 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":120,"threads_started":1}
I20260926 11:35:18.854892 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b02ccab9d8483a8585cb9e89cb9999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260926 11:35:18.855080 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:18.855161 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:18.855270 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:18.856530 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b02ccab9d8483a8585cb9e89cb9999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260926 11:35:18.856956 1554 leader_election.cc:302] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1b02ccab9d8483a8585cb9e89cb9999; no voters:
I20260926 11:35:18.857372 1554 leader_election.cc:288] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:18.857494 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:18.857707 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [term 1 LEADER]: Becoming Leader. State: Replica: a1b02ccab9d8483a8585cb9e89cb9999, State: Running, Role: LEADER
I20260926 11:35:18.858042 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b02ccab9d8483a8585cb9e89cb9999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260926 11:35:18.858740 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:18.859894 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b02ccab9d8483a8585cb9e89cb9999. Latest consensus state: current_term: 1 leader_uuid: "a1b02ccab9d8483a8585cb9e89cb9999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b02ccab9d8483a8585cb9e89cb9999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } } }
I20260926 11:35:18.860035 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:18.860312 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b02ccab9d8483a8585cb9e89cb9999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b02ccab9d8483a8585cb9e89cb9999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } } }
I20260926 11:35:18.860406 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:18.860572 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:18.863886 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:18.870358 1568 catalog_manager.cc:1347] Generated new cluster ID: d912e497496b47acb0410ba58ef97fcc
I20260926 11:35:18.870452 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:18.887516 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:18.888867 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:18.900312 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999: Generated new TSK 0
I20260926 11:35:18.901341 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:18.939188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36489
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:19.061838 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:19.062155 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:19.062399 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:19.066953 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:19.067034 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:19.067077 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:35:19.067113 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:19.073549 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:19.074747 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:19.075907 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:19.087081 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:19.087106 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:19.087136 1580 server_base.cc:1034] running on GCE node
W20260926 11:35:19.087081 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:19.087965 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:19.088610 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:19.089797 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790422519089780 us; error 30 us; skew 500 ppm
Sep 26 11:35:19 dist-test-slave-4pw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:19.093391 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:19.094444 1580 webserver.cc:460] Webserver started at http://127.0.98.1:45173/ using document root <none> and password file <none>
I20260926 11:35:19.095052 1580 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:19.095116 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:19.095347 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:19.097553 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "2108b87bbb26492782c29943d63cbf57"
format_stamp: "Formatted at 2026-09-26 11:35:19 on dist-test-slave-4pw5"
server_key: "eb2eb15913ac5de6b5d43c5d4a592ade"
server_key_iv: "47d29cd2b6620ff3e8da21259f7b1230"
server_key_version: "encryptionkey@0"
I20260926 11:35:19.098410 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2108b87bbb26492782c29943d63cbf57"
format_stamp: "Formatted at 2026-09-26 11:35:19 on dist-test-slave-4pw5"
server_key: "eb2eb15913ac5de6b5d43c5d4a592ade"
server_key_iv: "47d29cd2b6620ff3e8da21259f7b1230"
server_key_version: "encryptionkey@0"
I20260926 11:35:19.102541 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:35:19.105259 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:19.106971 1580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:35:19.107087 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "2108b87bbb26492782c29943d63cbf57"
format_stamp: "Formatted at 2026-09-26 11:35:19 on dist-test-slave-4pw5"
server_key: "eb2eb15913ac5de6b5d43c5d4a592ade"
server_key_iv: "47d29cd2b6620ff3e8da21259f7b1230"
server_key_version: "encryptionkey@0"
I20260926 11:35:19.107266 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:19.135196 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:19.138687 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:19.138926 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:19.140611 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:19.140697 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:19.140775 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:19.140822 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:35:19 dist-test-slave-4pw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:19.152195 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40451
I20260926 11:35:19.152233 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40451 every 8 connection(s)
I20260926 11:35:19.153438 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:19.155064 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.141412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56393 (local address 127.0.98.62:36489)
0926 11:35:19.141587 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.141593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.142498 (+ 905us) server_negotiation.cc:407] Connection header received
0926 11:35:19.143710 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.143715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.143797 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.143857 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:19.145542 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.146198 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.147450 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.147786 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.147911 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.150727 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:19.150734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:19.150740 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:19.150776 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:19.152598 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.153127 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.153131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.153133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.153228 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.153713 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.153717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.153718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.153988 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:19.154094 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.154786 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.154930 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:35:19.155961 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.141852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.98.1:56393)
0926 11:35:19.142318 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.142369 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.143415 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.144207 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.144214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.144721 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:19.145391 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.145407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.146338 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.146342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.147313 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.147323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.147960 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.147964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.148132 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.148808 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:19.148826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:19.150528 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:19.152723 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.152729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.152741 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.153009 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.153369 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.153372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.153374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.153579 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.154151 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:19.154160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:19.154515 (+ 355us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.154683 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.154961 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":133,"threads_started":1}
I20260926 11:35:19.157126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 1580
I20260926 11:35:19.157266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:19.157545 392 external_mini_cluster.cc:1428] Setting key c1049b73398677cc9ffe1677607300f4
I20260926 11:35:19.157626 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36489
I20260926 11:35:19.157874 1711 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:19.158527 1711 heartbeater.cc:511] Master 127.0.98.62:36489 requested a full tablet report, sending...
I20260926 11:35:19.160387 1508 ts_manager.cc:194] Registered new tserver with Master: 2108b87bbb26492782c29943d63cbf57 (127.0.98.1:40451)
I20260926 11:35:19.162007 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56393
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:35:19.170086 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.160508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.98.1:54911)
0926 11:35:19.160870 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.160888 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.161008 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.161450 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.161454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.161778 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:19.161949 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.161960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.163104 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.163107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.164066 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.164077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.164809 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.164816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.164919 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.165605 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:19.165617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:19.166077 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:19.168460 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.168463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.168465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.168690 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.169028 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.169030 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.169031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.169115 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.169628 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:19.169630 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:19.169721 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.169806 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.169924 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":140,"threads_started":1}
I20260926 11:35:19.170107 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.160562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54911 (local address 127.0.98.62:36489)
0926 11:35:19.160707 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.160712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.160954 (+ 242us) server_negotiation.cc:407] Connection header received
0926 11:35:19.161143 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.161148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.161207 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.161275 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:19.162093 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.162964 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.164231 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.164662 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.164803 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.166262 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:19.166267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:19.166271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:19.166301 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:19.168309 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.168810 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.168817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.168821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.168922 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.169220 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.169225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.169227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.169510 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:19.169628 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.169879 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.169987 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:35:19.192204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36489
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:19.319706 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:19.320009 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:19.320057 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:19.324383 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:19.324462 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:19.324504 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:35:19.324569 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:19.330260 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:19.331427 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:19.332695 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:19.343211 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:19.343204 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:19.343211 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:19.343751 1720 server_base.cc:1034] running on GCE node
I20260926 11:35:19.344273 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:19.345005 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:19.346220 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790422519346192 us; error 38 us; skew 500 ppm
Sep 26 11:35:19 dist-test-slave-4pw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:19.349695 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:35:19.350749 1720 webserver.cc:460] Webserver started at http://127.0.98.2:34035/ using document root <none> and password file <none>
I20260926 11:35:19.351413 1720 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:19.351482 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:19.351753 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:19.354113 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "d116ea0f6bba4e18ba432dc9e2564531"
format_stamp: "Formatted at 2026-09-26 11:35:19 on dist-test-slave-4pw5"
server_key: "bf428604053396116c9c281865effb1c"
server_key_iv: "d75679704ffb83383fc5cb61c3135a86"
server_key_version: "encryptionkey@0"
I20260926 11:35:19.354768 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d116ea0f6bba4e18ba432dc9e2564531"
format_stamp: "Formatted at 2026-09-26 11:35:19 on dist-test-slave-4pw5"
server_key: "bf428604053396116c9c281865effb1c"
server_key_iv: "d75679704ffb83383fc5cb61c3135a86"
server_key_version: "encryptionkey@0"
I20260926 11:35:19.359549 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:35:19.362699 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:19.364334 1720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 11:35:19.364530 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "d116ea0f6bba4e18ba432dc9e2564531"
format_stamp: "Formatted at 2026-09-26 11:35:19 on dist-test-slave-4pw5"
server_key: "bf428604053396116c9c281865effb1c"
server_key_iv: "d75679704ffb83383fc5cb61c3135a86"
server_key_version: "encryptionkey@0"
I20260926 11:35:19.364672 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:19.384161 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:19.387830 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:19.388067 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:19.389647 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:19.389730 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:19.389801 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:19.389844 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:35:19 dist-test-slave-4pw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:19.401666 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46039
I20260926 11:35:19.401783 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46039 every 8 connection(s)
I20260926 11:35:19.402928 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:19.405277 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.390890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32921 (local address 127.0.98.62:36489)
0926 11:35:19.391050 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.391054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.392204 (+ 1150us) server_negotiation.cc:407] Connection header received
0926 11:35:19.393606 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.393611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.393668 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.393762 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:19.395389 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.396435 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.397742 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.397981 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.398099 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.401006 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:19.401009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:19.401011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:19.401037 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:19.402689 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.403300 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.403303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.403305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.403416 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.403869 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.403872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.403874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.404115 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:19.404200 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.404966 (+ 766us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.405146 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 11:35:19.406029 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.391462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.98.2:32921)
0926 11:35:19.392014 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.392058 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.393324 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.393935 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.393945 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.394431 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:19.395197 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.395216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.396601 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.396605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.397592 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.397602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.398155 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.398160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.398336 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.398972 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:19.398989 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:19.400815 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:19.402817 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.402831 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.402846 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.403189 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.403527 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.403530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.403532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.403720 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.404223 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:19.404228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:19.404559 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.404801 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.405082 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":136,"threads_started":1}
I20260926 11:35:19.407626 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36489
I20260926 11:35:19.407826 1851 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:19.408442 1851 heartbeater.cc:511] Master 127.0.98.62:36489 requested a full tablet report, sending...
I20260926 11:35:19.409718 1507 ts_manager.cc:194] Registered new tserver with Master: d116ea0f6bba4e18ba432dc9e2564531 (127.0.98.2:46039)
I20260926 11:35:19.410423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 1720
I20260926 11:35:19.410544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:19.410512 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32921
I20260926 11:35:19.410938 392 external_mini_cluster.cc:1428] Setting key 9568ac2e2f19bc3b46b602324fc5d136
I20260926 11:35:19.419867 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.409900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.98.2:44047)
0926 11:35:19.411052 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.411068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.411182 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.411466 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.411469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.411759 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:19.411943 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.411952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.413014 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.413017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.413811 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.413821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.414322 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.414325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.414438 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.415070 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:19.415090 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:19.415613 (+ 523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:19.417831 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.417834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.417836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.418141 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.418512 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.418515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.418516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.418615 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.419290 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:19.419293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:19.419386 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.419487 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.419713 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":118,"threads_started":1}
I20260926 11:35:19.419974 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.410010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44047 (local address 127.0.98.62:36489)
0926 11:35:19.410137 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.410142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.411158 (+ 1016us) server_negotiation.cc:407] Connection header received
0926 11:35:19.411268 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.411271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.411322 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.411446 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:19.412056 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.412867 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.413948 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.414197 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.414320 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.415784 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:19.415787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:19.415789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:19.415814 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:19.417689 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.418252 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.418258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.418262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.418368 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.418750 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.418756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.418760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.419119 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:19.419247 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.419598 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.419746 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:35:19.445014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36489
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:19.563349 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:19.563685 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:19.563738 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:19.568005 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:19.568125 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:19.568204 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:35:19.568279 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:19.574292 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:19.575488 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:19.576623 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:19.587379 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:19.587529 1860 server_base.cc:1034] running on GCE node
W20260926 11:35:19.587371 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:19.587340 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:19.588182 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:19.588879 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:19.590088 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790422519590060 us; error 44 us; skew 500 ppm
Sep 26 11:35:19 dist-test-slave-4pw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:19.593204 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:35:19.594342 1860 webserver.cc:460] Webserver started at http://127.0.98.3:37549/ using document root <none> and password file <none>
I20260926 11:35:19.594997 1860 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:19.595069 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:19.595260 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:19.597705 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "d0867a2a7055483f9a7d4f37e4614874"
format_stamp: "Formatted at 2026-09-26 11:35:19 on dist-test-slave-4pw5"
server_key: "c7f8f9759de0b58aaf22948f41694eed"
server_key_iv: "6ac33a8e259f4138b27364fb68c1377c"
server_key_version: "encryptionkey@0"
I20260926 11:35:19.598315 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d0867a2a7055483f9a7d4f37e4614874"
format_stamp: "Formatted at 2026-09-26 11:35:19 on dist-test-slave-4pw5"
server_key: "c7f8f9759de0b58aaf22948f41694eed"
server_key_iv: "6ac33a8e259f4138b27364fb68c1377c"
server_key_version: "encryptionkey@0"
I20260926 11:35:19.602794 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:35:19.605821 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:19.607280 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 11:35:19.607463 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "d0867a2a7055483f9a7d4f37e4614874"
format_stamp: "Formatted at 2026-09-26 11:35:19 on dist-test-slave-4pw5"
server_key: "c7f8f9759de0b58aaf22948f41694eed"
server_key_iv: "6ac33a8e259f4138b27364fb68c1377c"
server_key_version: "encryptionkey@0"
I20260926 11:35:19.607664 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:19.619122 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:19.623363 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:19.623660 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:19.625578 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:19.625638 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:19.625682 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:19.625698 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:35:19 dist-test-slave-4pw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:19.637243 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40943
I20260926 11:35:19.637270 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40943 every 8 connection(s)
I20260926 11:35:19.638239 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:19.641043 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.626435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54595 (local address 127.0.98.62:36489)
0926 11:35:19.626613 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.626618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.627899 (+ 1281us) server_negotiation.cc:407] Connection header received
0926 11:35:19.629362 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.629366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.629436 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.629564 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:19.631064 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.631774 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.633358 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.633611 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.633760 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.636967 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:19.636972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:19.636975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:19.637012 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:19.638937 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.639523 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.639526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.639527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.639666 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.640027 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.640030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.640031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.640235 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:19.640318 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.640807 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.640915 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 11:35:19.641929 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.626959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.98.3:54595)
0926 11:35:19.627611 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.627674 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.629069 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.629774 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.629782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.630293 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:19.630890 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.630906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.631918 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.631922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.633218 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.633228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.633822 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.633826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.634031 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.634735 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:19.634754 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:19.636742 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:19.639065 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.639070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.639085 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.639421 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.639805 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.639807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.639808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.639939 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.640343 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:19.640349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:19.640586 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.640773 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.641036 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":133,"threads_started":1}
I20260926 11:35:19.642721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 1860
I20260926 11:35:19.642863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:19.643150 392 external_mini_cluster.cc:1428] Setting key edd2d35fb7ca9fa08508bea56b4364c7
I20260926 11:35:19.643478 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36489
I20260926 11:35:19.643671 1991 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:19.644328 1991 heartbeater.cc:511] Master 127.0.98.62:36489 requested a full tablet report, sending...
I20260926 11:35:19.645373 1508 ts_manager.cc:194] Registered new tserver with Master: d0867a2a7055483f9a7d4f37e4614874 (127.0.98.3:40943)
I20260926 11:35:19.646343 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54595
I20260926 11:35:19.661038 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.646283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47433 (local address 127.0.98.62:36489)
0926 11:35:19.646462 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.646470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.647379 (+ 909us) server_negotiation.cc:407] Connection header received
0926 11:35:19.647472 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.647475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.647526 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.647637 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:19.648322 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.649187 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.650271 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.650618 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.650996 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.655249 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:19.655257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:19.655264 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:19.655301 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:19.658164 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.658783 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.658789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.658792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.658905 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.659608 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.659615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.659617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.659892 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:19.659984 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.660440 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.660824 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:35:19.661657 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.646567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.98.3:47433)
0926 11:35:19.647231 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.647251 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.647371 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.647767 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.647770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.648022 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:19.648144 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.648152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.649354 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.649358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.650125 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.650135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.651020 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.651024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.651151 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.654341 (+ 3190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:19.654359 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:19.655049 (+ 690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:19.658331 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.658337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.658340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.658661 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.659075 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:19.659079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:19.659081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:19.659450 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:19.660072 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:19.660075 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:19.660193 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.661204 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.661390 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":150,"threads_started":1}
I20260926 11:35:19.661685 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 11:35:19 dist-test-slave-4pw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:35:19 dist-test-slave-4pw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:19.698256 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.685586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.0.98.62:36489)
0926 11:35:19.685751 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.685757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.685776 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:35:19.685831 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.685834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.685918 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.686021 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:19.686999 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.687971 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.689396 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.689891 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.690048 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.693815 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:19.693820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:19.693823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:19.693864 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:19.695972 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.696582 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.696586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.696591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.696716 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:19.697083 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:19.697086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:19.697088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:19.697373 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:19.697545 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.697894 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.698035 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 11:35:19.702706 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:19.707535 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:19.725033 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.718481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.0.98.3:40943)
0926 11:35:19.719265 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.719281 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.719307 (+ 26us) server_negotiation.cc:407] Connection header received
0926 11:35:19.719392 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.719401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.719683 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.720276 (+ 593us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:19.720429 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.721600 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.723376 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.724182 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.724326 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.724522 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.724753 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.724863 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":122,"threads_started":1}
I20260926 11:35:19.725091 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.719082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40451 (local address 127.0.0.1:42642)
0926 11:35:19.719840 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.719857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.719953 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.720759 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.720762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.720780 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:19.720892 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.720899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.722080 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.722086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.723299 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.723311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.724474 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.724479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.724563 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.724582 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.724683 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.724845 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":5,"thread_start_us":111,"threads_started":1}
I20260926 11:35:19.725102 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.718313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40943 (local address 127.0.0.1:56688)
0926 11:35:19.718695 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.718755 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.718916 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.719885 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.719891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.719920 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:19.720100 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.720109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.721766 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.721769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.723172 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.723182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.724347 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.724353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.724523 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.724575 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.724704 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.724845 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":120,"spinlock_wait_cycles":768,"thread_start_us":77,"threads_started":1}
I20260926 11:35:19.726042 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.719295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.0.98.1:40451)
0926 11:35:19.720308 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.720316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.720342 (+ 26us) server_negotiation.cc:407] Connection header received
0926 11:35:19.720423 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.720428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.720619 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.720787 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:19.720999 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.721959 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.723462 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.724330 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.725577 (+ 1247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.725750 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.725802 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.725899 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":109,"threads_started":1}
I20260926 11:35:19.727438 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.719688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.0.98.2:46039)
0926 11:35:19.720240 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.720249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.720477 (+ 228us) server_negotiation.cc:407] Connection header received
0926 11:35:19.720654 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.720660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.720876 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.721031 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:19.722402 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.723859 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.725826 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.726847 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.726980 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.727146 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.727206 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.727296 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":111,"threads_started":1}
I20260926 11:35:19.727427 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.719466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46039 (local address 127.0.0.1:55782)
0926 11:35:19.720371 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.720383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.720500 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.722111 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.722115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.722135 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:19.722255 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.722262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.724015 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.724020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.725631 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.725641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.726976 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.726980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.727074 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.727099 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.727176 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.727274 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":6,"thread_start_us":128,"threads_started":1}
I20260926 11:35:19.729225 1925 tablet_service.cc:1467] Processing CreateTablet for tablet f48fd392d8ec4497ab4390889e22de31 (DEFAULT_TABLE table=test-table [id=d7c69eca49c841b7bd41ef518f990559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:19.729223 1645 tablet_service.cc:1467] Processing CreateTablet for tablet f48fd392d8ec4497ab4390889e22de31 (DEFAULT_TABLE table=test-table [id=d7c69eca49c841b7bd41ef518f990559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:19.730509 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f48fd392d8ec4497ab4390889e22de31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:19.730597 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f48fd392d8ec4497ab4390889e22de31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:19.730542 1785 tablet_service.cc:1467] Processing CreateTablet for tablet f48fd392d8ec4497ab4390889e22de31 (DEFAULT_TABLE table=test-table [id=d7c69eca49c841b7bd41ef518f990559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:19.731724 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f48fd392d8ec4497ab4390889e22de31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:19.738947 2014 tablet_bootstrap.cc:492] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: Bootstrap starting.
I20260926 11:35:19.739500 2015 tablet_bootstrap.cc:492] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: Bootstrap starting.
I20260926 11:35:19.740314 2016 tablet_bootstrap.cc:492] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: Bootstrap starting.
I20260926 11:35:19.742241 2015 tablet_bootstrap.cc:654] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:19.742331 2016 tablet_bootstrap.cc:654] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:19.743167 2016 log.cc:824] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:19.743177 2015 log.cc:824] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:19.745532 2016 tablet_bootstrap.cc:492] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: No bootstrap required, opened a new log
I20260926 11:35:19.745774 2016 ts_tablet_manager.cc:1397] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 11:35:19.745919 2014 tablet_bootstrap.cc:654] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:19.746752 2014 log.cc:824] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:19.749141 2014 tablet_bootstrap.cc:492] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: No bootstrap required, opened a new log
I20260926 11:35:19.749147 2015 tablet_bootstrap.cc:492] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: No bootstrap required, opened a new log
I20260926 11:35:19.749367 2015 ts_tablet_manager.cc:1397] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260926 11:35:19.749367 2014 ts_tablet_manager.cc:1397] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260926 11:35:19.749993 2016 raft_consensus.cc:348] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } }
I20260926 11:35:19.750286 2016 raft_consensus.cc:374] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:19.750347 2016 raft_consensus.cc:729] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d116ea0f6bba4e18ba432dc9e2564531, State: Initialized, Role: FOLLOWER
I20260926 11:35:19.750828 2016 consensus_queue.cc:260] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } }
I20260926 11:35:19.751633 1851 heartbeater.cc:503] Master 127.0.98.62:36489 was elected leader, sending a full tablet report...
I20260926 11:35:19.752029 2016 ts_tablet_manager.cc:1428] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260926 11:35:19.754122 2015 raft_consensus.cc:348] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } }
I20260926 11:35:19.754462 2015 raft_consensus.cc:374] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:19.754522 2015 raft_consensus.cc:729] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0867a2a7055483f9a7d4f37e4614874, State: Initialized, Role: FOLLOWER
I20260926 11:35:19.754401 2014 raft_consensus.cc:348] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } }
I20260926 11:35:19.754681 2014 raft_consensus.cc:374] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:19.754751 2014 raft_consensus.cc:729] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2108b87bbb26492782c29943d63cbf57, State: Initialized, Role: FOLLOWER
I20260926 11:35:19.755134 2014 consensus_queue.cc:260] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } }
I20260926 11:35:19.755095 2015 consensus_queue.cc:260] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } }
I20260926 11:35:19.756264 2015 ts_tablet_manager.cc:1428] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 11:35:19.756264 2014 ts_tablet_manager.cc:1428] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260926 11:35:19.756685 1711 heartbeater.cc:503] Master 127.0.98.62:36489 was elected leader, sending a full tablet report...
I20260926 11:35:19.757038 1991 heartbeater.cc:503] Master 127.0.98.62:36489 was elected leader, sending a full tablet report...
I20260926 11:35:19.779834 2021 raft_consensus.cc:482] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:19.780114 2021 raft_consensus.cc:504] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } }
I20260926 11:35:19.781435 2021 leader_election.cc:288] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0867a2a7055483f9a7d4f37e4614874 (127.0.98.3:40943), d116ea0f6bba4e18ba432dc9e2564531 (127.0.98.2:46039)
I20260926 11:35:19.786275 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.781759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40943 (local address 127.0.98.1:51053)
0926 11:35:19.782222 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.782234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.782508 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.782942 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.782945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.782960 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:19.783152 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.783164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.784064 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.784069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.785059 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.785068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.785770 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.785773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.785921 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.785942 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.786003 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.786086 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":119,"spinlock_wait_cycles":7936,"thread_start_us":136,"threads_started":1}
I20260926 11:35:19.786271 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.781984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51053 (local address 127.0.98.3:40943)
0926 11:35:19.782108 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.782113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.782445 (+ 332us) server_negotiation.cc:407] Connection header received
0926 11:35:19.782675 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.782682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.782739 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.782787 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:19.783297 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.783956 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.785182 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.785634 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.785770 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.785840 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.786045 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.786128 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 11:35:19.786366 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.781908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35879 (local address 127.0.98.2:46039)
0926 11:35:19.782055 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.782071 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.782571 (+ 500us) server_negotiation.cc:407] Connection header received
0926 11:35:19.782753 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.782756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.782815 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.782918 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:19.783382 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.784021 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.785256 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.785633 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.785746 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.785809 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.786108 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.786235 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:35:19.786309 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.781767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46039 (local address 127.0.98.1:35879)
0926 11:35:19.782406 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:19.782420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:19.782598 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:19.782929 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:19.782932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:19.782953 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:19.783250 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.783261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.784147 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.784149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.785090 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:19.785107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.785773 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.785777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:19.785954 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.785972 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 11:35:19.786075 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 11:35:19.786140 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":143,"thread_start_us":111,"threads_started":1}
I20260926 11:35:19.787063 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48fd392d8ec4497ab4390889e22de31" candidate_uuid: "2108b87bbb26492782c29943d63cbf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0867a2a7055483f9a7d4f37e4614874" is_pre_election: true
I20260926 11:35:19.787146 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48fd392d8ec4497ab4390889e22de31" candidate_uuid: "2108b87bbb26492782c29943d63cbf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d116ea0f6bba4e18ba432dc9e2564531" is_pre_election: true
I20260926 11:35:19.787410 1945 raft_consensus.cc:2454] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2108b87bbb26492782c29943d63cbf57 in term 0.
I20260926 11:35:19.787516 1805 raft_consensus.cc:2454] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2108b87bbb26492782c29943d63cbf57 in term 0.
I20260926 11:35:19.788087 1598 leader_election.cc:302] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2108b87bbb26492782c29943d63cbf57, d0867a2a7055483f9a7d4f37e4614874, d116ea0f6bba4e18ba432dc9e2564531; no voters:
I20260926 11:35:19.788441 2021 raft_consensus.cc:2781] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:19.788518 2021 raft_consensus.cc:482] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:19.788548 2021 raft_consensus.cc:3037] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:19.789614 2021 raft_consensus.cc:504] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } }
I20260926 11:35:19.789974 2021 leader_election.cc:288] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [CANDIDATE]: Term 1 election: Requested vote from peers d0867a2a7055483f9a7d4f37e4614874 (127.0.98.3:40943), d116ea0f6bba4e18ba432dc9e2564531 (127.0.98.2:46039)
I20260926 11:35:19.790426 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48fd392d8ec4497ab4390889e22de31" candidate_uuid: "2108b87bbb26492782c29943d63cbf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d116ea0f6bba4e18ba432dc9e2564531"
I20260926 11:35:19.790426 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48fd392d8ec4497ab4390889e22de31" candidate_uuid: "2108b87bbb26492782c29943d63cbf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0867a2a7055483f9a7d4f37e4614874"
I20260926 11:35:19.790563 1805 raft_consensus.cc:3037] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:19.790563 1945 raft_consensus.cc:3037] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:19.792476 1945 raft_consensus.cc:2454] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2108b87bbb26492782c29943d63cbf57 in term 1.
I20260926 11:35:19.792476 1805 raft_consensus.cc:2454] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2108b87bbb26492782c29943d63cbf57 in term 1.
I20260926 11:35:19.792857 1597 leader_election.cc:302] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2108b87bbb26492782c29943d63cbf57, d116ea0f6bba4e18ba432dc9e2564531; no voters:
I20260926 11:35:19.793079 2021 raft_consensus.cc:2781] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:19.793367 2021 raft_consensus.cc:686] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 1 LEADER]: Becoming Leader. State: Replica: 2108b87bbb26492782c29943d63cbf57, State: Running, Role: LEADER
I20260926 11:35:19.793726 2021 consensus_queue.cc:237] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } }
I20260926 11:35:19.797073 1507 catalog_manager.cc:5495] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2108b87bbb26492782c29943d63cbf57 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2108b87bbb26492782c29943d63cbf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:19.818857 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:19.815125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.0.98.1:40451)
0926 11:35:19.815310 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:19.815321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:19.815358 (+ 37us) server_negotiation.cc:407] Connection header received
0926 11:35:19.815433 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:19.815435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:19.815500 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:19.815661 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:19.815909 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.816455 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.817300 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:19.817558 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:19.817659 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:19.817974 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:19.818487 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:19.818588 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:19.818676 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 11:35:19.818734 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 11:35:19.825232 1805 raft_consensus.cc:1261] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [term 1 FOLLOWER]: Refusing update from remote peer 2108b87bbb26492782c29943d63cbf57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:19.825232 1945 raft_consensus.cc:1261] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Refusing update from remote peer 2108b87bbb26492782c29943d63cbf57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:19.826000 2021 consensus_queue.cc:1035] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:19.826294 2025 consensus_queue.cc:1035] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:19.839401 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333570641195663360 to 7333570641078001664. Current Snapshot: MvccSnapshot[applied={T|T < 7333570641195663360}]
I20260926 11:35:19.840560 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333570641195663360 to 7333570641078001664. Current Snapshot: MvccSnapshot[applied={T|T < 7333570641195663360}]
I20260926 11:35:19.842105 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333570641195663360 to 7333570641078001664. Current Snapshot: MvccSnapshot[applied={T|T < 7333570641195663360}]
I20260926 11:35:19.858443 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:19.858775 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:19.861866 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999: Sending DeleteTablet for 3 replicas of tablet f48fd392d8ec4497ab4390889e22de31
I20260926 11:35:19.862844 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet f48fd392d8ec4497ab4390889e22de31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55782
I20260926 11:35:19.862910 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet f48fd392d8ec4497ab4390889e22de31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56688
I20260926 11:35:19.862907 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet f48fd392d8ec4497ab4390889e22de31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260926 11:35:19.863539 2039 tablet_replica.cc:330] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: stopping tablet replica
I20260926 11:35:19.863780 2038 tablet_replica.cc:330] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: stopping tablet replica
I20260926 11:35:19.864094 2039 raft_consensus.cc:2229] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:19.864108 2038 raft_consensus.cc:2229] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:19.864463 2038 raft_consensus.cc:2258] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:19.864589 2039 raft_consensus.cc:2258] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:19.864475 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 11:35:19.864910 2040 tablet_replica.cc:330] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: stopping tablet replica
W20260926 11:35:19.865028 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:19.865199 2040 raft_consensus.cc:2229] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:19.865607 2040 raft_consensus.cc:2258] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:19.867055 2040 ts_tablet_manager.cc:1905] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:19.867055 2038 ts_tablet_manager.cc:1905] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:19.867055 2039 ts_tablet_manager.cc:1905] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:19.870100 2040 ts_tablet_manager.cc:1918] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:19.870218 2040 log.cc:1196] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/f48fd392d8ec4497ab4390889e22de31
I20260926 11:35:19.870571 2040 ts_tablet_manager.cc:1939] T f48fd392d8ec4497ab4390889e22de31 P 2108b87bbb26492782c29943d63cbf57: Deleting consensus metadata
I20260926 11:35:19.871032 2038 ts_tablet_manager.cc:1918] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:19.871016 1645 tablet_service.cc:1467] Processing CreateTablet for tablet f33ba5210936407bbb6e4f926da6d76e (DEFAULT_TABLE table=test-table [id=492ab2ce044248468bf393bda0c53f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:19.871134 2038 log.cc:1196] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/f48fd392d8ec4497ab4390889e22de31
I20260926 11:35:19.871312 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f33ba5210936407bbb6e4f926da6d76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:19.871464 2038 ts_tablet_manager.cc:1939] T f48fd392d8ec4497ab4390889e22de31 P d116ea0f6bba4e18ba432dc9e2564531: Deleting consensus metadata
I20260926 11:35:19.871479 1785 tablet_service.cc:1467] Processing CreateTablet for tablet f33ba5210936407bbb6e4f926da6d76e (DEFAULT_TABLE table=test-table [id=492ab2ce044248468bf393bda0c53f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:19.871771 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f33ba5210936407bbb6e4f926da6d76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:19.872084 1494 catalog_manager.cc:4841] TS 2108b87bbb26492782c29943d63cbf57 (127.0.98.1:40451): tablet f48fd392d8ec4497ab4390889e22de31 (table test-table [id=d7c69eca49c841b7bd41ef518f990559]) successfully deleted
I20260926 11:35:19.873986 2014 tablet_bootstrap.cc:492] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: Bootstrap starting.
I20260926 11:35:19.874164 2016 tablet_bootstrap.cc:492] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: Bootstrap starting.
I20260926 11:35:19.874958 1925 tablet_service.cc:1467] Processing CreateTablet for tablet f33ba5210936407bbb6e4f926da6d76e (DEFAULT_TABLE table=test-table [id=492ab2ce044248468bf393bda0c53f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:19.875113 2014 tablet_bootstrap.cc:654] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:19.875133 2016 tablet_bootstrap.cc:654] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:19.875240 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f33ba5210936407bbb6e4f926da6d76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:19.876050 1493 catalog_manager.cc:4841] TS d116ea0f6bba4e18ba432dc9e2564531 (127.0.98.2:46039): tablet f48fd392d8ec4497ab4390889e22de31 (table test-table [id=d7c69eca49c841b7bd41ef518f990559]) successfully deleted
I20260926 11:35:19.876927 2016 tablet_bootstrap.cc:492] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: No bootstrap required, opened a new log
I20260926 11:35:19.877017 2016 ts_tablet_manager.cc:1397] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:19.878134 2016 raft_consensus.cc:348] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:19.878324 2016 raft_consensus.cc:374] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:19.878398 2016 raft_consensus.cc:729] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d116ea0f6bba4e18ba432dc9e2564531, State: Initialized, Role: FOLLOWER
I20260926 11:35:19.878634 2015 tablet_bootstrap.cc:492] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: Bootstrap starting.
I20260926 11:35:19.878640 2016 consensus_queue.cc:260] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:19.878973 2016 ts_tablet_manager.cc:1428] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:35:19.879674 2015 tablet_bootstrap.cc:654] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:19.882050 2014 tablet_bootstrap.cc:492] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: No bootstrap required, opened a new log
I20260926 11:35:19.882181 2014 ts_tablet_manager.cc:1397] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260926 11:35:19.882421 2015 tablet_bootstrap.cc:492] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: No bootstrap required, opened a new log
I20260926 11:35:19.882532 2015 ts_tablet_manager.cc:1397] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 11:35:19.882761 2014 raft_consensus.cc:348] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:19.882890 2014 raft_consensus.cc:374] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:19.882932 2014 raft_consensus.cc:729] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2108b87bbb26492782c29943d63cbf57, State: Initialized, Role: FOLLOWER
I20260926 11:35:19.883101 2039 ts_tablet_manager.cc:1918] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:19.883116 2014 consensus_queue.cc:260] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:19.883073 2015 raft_consensus.cc:348] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:19.883244 2015 raft_consensus.cc:374] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:19.883284 2015 raft_consensus.cc:729] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0867a2a7055483f9a7d4f37e4614874, State: Initialized, Role: FOLLOWER
I20260926 11:35:19.883410 2015 consensus_queue.cc:260] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:19.883513 2014 ts_tablet_manager.cc:1428] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 11:35:19.883708 2015 ts_tablet_manager.cc:1428] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 11:35:19.884146 2039 log.cc:1196] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/f48fd392d8ec4497ab4390889e22de31
I20260926 11:35:19.884572 2039 ts_tablet_manager.cc:1939] T f48fd392d8ec4497ab4390889e22de31 P d0867a2a7055483f9a7d4f37e4614874: Deleting consensus metadata
I20260926 11:35:19.885931 1494 catalog_manager.cc:4841] TS d0867a2a7055483f9a7d4f37e4614874 (127.0.98.3:40943): tablet f48fd392d8ec4497ab4390889e22de31 (table test-table [id=d7c69eca49c841b7bd41ef518f990559]) successfully deleted
W20260926 11:35:19.889286 1992 tablet.cc:2369] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:19.905020 1712 tablet.cc:2369] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:19.905918 1852 tablet.cc:2369] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:35:20.103549 2020 raft_consensus.cc:482] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:20.103752 2020 raft_consensus.cc:504] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.105031 2020 leader_election.cc:288] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2108b87bbb26492782c29943d63cbf57 (127.0.98.1:40451), d0867a2a7055483f9a7d4f37e4614874 (127.0.98.3:40943)
I20260926 11:35:20.109786 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.105857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34915 (local address 127.0.98.3:40943)
0926 11:35:20.106008 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:20.106013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:20.106092 (+ 79us) server_negotiation.cc:407] Connection header received
0926 11:35:20.106228 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:20.106232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:20.106329 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:20.106406 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:20.106800 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.107391 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.108765 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.109148 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.109227 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.109351 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:20.109569 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 11:35:20.109647 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:35:20.109786 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.105737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40943 (local address 127.0.98.2:34915)
0926 11:35:20.106029 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:20.106042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:20.106128 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:20.106549 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:20.106552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:20.106571 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:20.106667 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.106673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.107471 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.107474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.108610 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.108618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.109277 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.109283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.109406 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.109471 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 11:35:20.109530 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 11:35:20.109616 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":48,"thread_start_us":94,"threads_started":1}
I20260926 11:35:20.109961 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.105409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43061 (local address 127.0.98.1:40451)
0926 11:35:20.105585 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:20.105590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:20.105723 (+ 133us) server_negotiation.cc:407] Connection header received
0926 11:35:20.105886 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:20.105889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:20.105938 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:20.106010 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:20.106349 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.107077 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.108785 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.109215 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.109325 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.109391 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:20.109752 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 11:35:20.109842 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 11:35:20.109961 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.105329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40451 (local address 127.0.98.2:43061)
0926 11:35:20.105603 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:20.105619 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:20.105773 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:20.106046 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:20.106050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:20.106073 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:20.106200 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.106217 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.107249 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.107252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.108607 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.108618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.109365 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.109371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.109609 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.109632 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:35:20.109707 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 11:35:20.109773 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":70,"thread_start_us":97,"threads_started":1}
I20260926 11:35:20.110461 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f33ba5210936407bbb6e4f926da6d76e" candidate_uuid: "d116ea0f6bba4e18ba432dc9e2564531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0867a2a7055483f9a7d4f37e4614874" is_pre_election: true
I20260926 11:35:20.110481 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f33ba5210936407bbb6e4f926da6d76e" candidate_uuid: "d116ea0f6bba4e18ba432dc9e2564531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2108b87bbb26492782c29943d63cbf57" is_pre_election: true
I20260926 11:35:20.110631 1945 raft_consensus.cc:2454] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d116ea0f6bba4e18ba432dc9e2564531 in term 0.
I20260926 11:35:20.110703 1665 raft_consensus.cc:2454] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d116ea0f6bba4e18ba432dc9e2564531 in term 0.
I20260926 11:35:20.111042 1738 leader_election.cc:302] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0867a2a7055483f9a7d4f37e4614874, d116ea0f6bba4e18ba432dc9e2564531; no voters:
I20260926 11:35:20.111302 2020 raft_consensus.cc:2781] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:20.111366 2020 raft_consensus.cc:482] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:20.111405 2020 raft_consensus.cc:3037] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:20.112599 2020 raft_consensus.cc:504] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.112946 2020 leader_election.cc:288] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [CANDIDATE]: Term 1 election: Requested vote from peers 2108b87bbb26492782c29943d63cbf57 (127.0.98.1:40451), d0867a2a7055483f9a7d4f37e4614874 (127.0.98.3:40943)
I20260926 11:35:20.113381 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f33ba5210936407bbb6e4f926da6d76e" candidate_uuid: "d116ea0f6bba4e18ba432dc9e2564531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2108b87bbb26492782c29943d63cbf57"
I20260926 11:35:20.113458 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f33ba5210936407bbb6e4f926da6d76e" candidate_uuid: "d116ea0f6bba4e18ba432dc9e2564531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0867a2a7055483f9a7d4f37e4614874"
I20260926 11:35:20.113534 1665 raft_consensus.cc:3037] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:20.113621 1945 raft_consensus.cc:3037] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:20.114668 1665 raft_consensus.cc:2454] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d116ea0f6bba4e18ba432dc9e2564531 in term 1.
I20260926 11:35:20.114962 1945 raft_consensus.cc:2454] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d116ea0f6bba4e18ba432dc9e2564531 in term 1.
I20260926 11:35:20.114962 1738 leader_election.cc:302] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2108b87bbb26492782c29943d63cbf57, d116ea0f6bba4e18ba432dc9e2564531; no voters:
I20260926 11:35:20.115302 2020 raft_consensus.cc:2781] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:20.115530 2020 raft_consensus.cc:686] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 1 LEADER]: Becoming Leader. State: Replica: d116ea0f6bba4e18ba432dc9e2564531, State: Running, Role: LEADER
I20260926 11:35:20.115833 2020 consensus_queue.cc:237] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.118101 1506 catalog_manager.cc:5495] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 reported cstate change: term changed from 0 to 1, leader changed from <none> to d116ea0f6bba4e18ba432dc9e2564531 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d116ea0f6bba4e18ba432dc9e2564531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } health_report { overall_health: UNKNOWN } } }
Sep 26 11:35:20 dist-test-slave-4pw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:20.203486 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.193746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.0.1:58256)
0926 11:35:20.194070 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:20.194085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:20.194211 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:20.194564 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:20.194566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:20.195027 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:20.195126 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.195133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.196094 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.196096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.196670 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.196678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.197171 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.197174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.197278 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.197869 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:20.197881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:20.199509 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:20.201555 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:20.201569 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:20.201576 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:20.201843 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:20.202155 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:20.202158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:20.202160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:20.202257 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:20.202692 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:20.202703 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:20.202835 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 11:35:20.202977 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 11:35:20.203263 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":67,"thread_start_us":126,"threads_started":1}
I20260926 11:35:20.203485 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.193822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.0.98.62:36489)
0926 11:35:20.193962 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:20.193967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:20.194166 (+ 199us) server_negotiation.cc:407] Connection header received
0926 11:35:20.194372 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:20.194375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:20.194421 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:20.194508 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:20.195313 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.195981 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.196816 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.197047 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.197153 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.199714 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:20.199719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:20.199722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:20.199748 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:20.201385 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:20.201958 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:20.201964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:20.201966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:20.202048 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:20.202363 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:20.202367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:20.202369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:20.202568 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:20.202682 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:20.203024 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 11:35:20.203330 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":57}
I20260926 11:35:20.208510 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58256:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 11:35:20.209344 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:20.215096 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 07f20fcf4cca4a46bada745958682b34 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7a056409246469f995e434e6facda15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:35:20.215201 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 07f20fcf4cca4a46bada745958682b34 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7a056409246469f995e434e6facda15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:35:20.215289 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 07f20fcf4cca4a46bada745958682b34 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7a056409246469f995e434e6facda15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:35:20.215547 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f20fcf4cca4a46bada745958682b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:20.215682 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f20fcf4cca4a46bada745958682b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:20.215689 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f20fcf4cca4a46bada745958682b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:20.218266 2014 tablet_bootstrap.cc:492] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: Bootstrap starting.
I20260926 11:35:20.218735 2015 tablet_bootstrap.cc:492] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: Bootstrap starting.
I20260926 11:35:20.219269 2014 tablet_bootstrap.cc:654] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:20.219749 2015 tablet_bootstrap.cc:654] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:20.219753 2016 tablet_bootstrap.cc:492] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: Bootstrap starting.
I20260926 11:35:20.220677 2014 tablet_bootstrap.cc:492] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: No bootstrap required, opened a new log
I20260926 11:35:20.220788 2014 ts_tablet_manager.cc:1397] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:20.220803 2016 tablet_bootstrap.cc:654] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:20.221028 2015 tablet_bootstrap.cc:492] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: No bootstrap required, opened a new log
I20260926 11:35:20.221140 2015 ts_tablet_manager.cc:1397] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:35:20.221371 2014 raft_consensus.cc:348] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.221534 2014 raft_consensus.cc:374] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:20.221580 2014 raft_consensus.cc:729] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2108b87bbb26492782c29943d63cbf57, State: Initialized, Role: FOLLOWER
I20260926 11:35:20.221678 2015 raft_consensus.cc:348] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.221735 2014 consensus_queue.cc:260] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.221819 2015 raft_consensus.cc:374] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:20.221846 2015 raft_consensus.cc:729] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0867a2a7055483f9a7d4f37e4614874, State: Initialized, Role: FOLLOWER
I20260926 11:35:20.222059 2014 ts_tablet_manager.cc:1428] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:20.222117 2016 tablet_bootstrap.cc:492] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: No bootstrap required, opened a new log
I20260926 11:35:20.222040 2033 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.222198 2016 ts_tablet_manager.cc:1397] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:35:20.222187 2033 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:20.222501 2015 consensus_queue.cc:260] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.222703 2016 raft_consensus.cc:348] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.222818 2016 raft_consensus.cc:374] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:20.222774 2022 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.222853 2016 raft_consensus.cc:729] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d116ea0f6bba4e18ba432dc9e2564531, State: Initialized, Role: FOLLOWER
I20260926 11:35:20.222911 2022 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:20.222977 2016 consensus_queue.cc:260] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.223271 2016 ts_tablet_manager.cc:1428] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:35:20.223232 2020 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.223362 2020 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:20.223552 2015 ts_tablet_manager.cc:1428] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:35:20.438596 2033 raft_consensus.cc:482] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:20.438758 2033 raft_consensus.cc:504] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.439373 2033 leader_election.cc:288] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d116ea0f6bba4e18ba432dc9e2564531 (127.0.98.2:46039), d0867a2a7055483f9a7d4f37e4614874 (127.0.98.3:40943)
I20260926 11:35:20.439877 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f20fcf4cca4a46bada745958682b34" candidate_uuid: "2108b87bbb26492782c29943d63cbf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d116ea0f6bba4e18ba432dc9e2564531" is_pre_election: true
I20260926 11:35:20.439947 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f20fcf4cca4a46bada745958682b34" candidate_uuid: "2108b87bbb26492782c29943d63cbf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0867a2a7055483f9a7d4f37e4614874" is_pre_election: true
I20260926 11:35:20.440037 1805 raft_consensus.cc:2454] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2108b87bbb26492782c29943d63cbf57 in term 0.
I20260926 11:35:20.440157 1945 raft_consensus.cc:2454] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2108b87bbb26492782c29943d63cbf57 in term 0.
I20260926 11:35:20.440424 1597 leader_election.cc:302] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2108b87bbb26492782c29943d63cbf57, d116ea0f6bba4e18ba432dc9e2564531; no voters:
I20260926 11:35:20.440598 2033 raft_consensus.cc:2781] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:20.440657 2033 raft_consensus.cc:482] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:20.440698 2033 raft_consensus.cc:3037] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:20.441743 2033 raft_consensus.cc:504] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.442168 2033 leader_election.cc:288] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [CANDIDATE]: Term 1 election: Requested vote from peers d116ea0f6bba4e18ba432dc9e2564531 (127.0.98.2:46039), d0867a2a7055483f9a7d4f37e4614874 (127.0.98.3:40943)
I20260926 11:35:20.442634 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f20fcf4cca4a46bada745958682b34" candidate_uuid: "2108b87bbb26492782c29943d63cbf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d116ea0f6bba4e18ba432dc9e2564531"
I20260926 11:35:20.442830 1805 raft_consensus.cc:3037] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:20.442828 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f20fcf4cca4a46bada745958682b34" candidate_uuid: "2108b87bbb26492782c29943d63cbf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0867a2a7055483f9a7d4f37e4614874"
I20260926 11:35:20.443048 1945 raft_consensus.cc:3037] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:20.444212 1805 raft_consensus.cc:2454] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2108b87bbb26492782c29943d63cbf57 in term 1.
I20260926 11:35:20.444231 1945 raft_consensus.cc:2454] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2108b87bbb26492782c29943d63cbf57 in term 1.
I20260926 11:35:20.444572 1598 leader_election.cc:302] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2108b87bbb26492782c29943d63cbf57, d0867a2a7055483f9a7d4f37e4614874; no voters:
I20260926 11:35:20.444872 2033 raft_consensus.cc:2781] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:20.445062 2033 raft_consensus.cc:686] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [term 1 LEADER]: Becoming Leader. State: Replica: 2108b87bbb26492782c29943d63cbf57, State: Running, Role: LEADER
I20260926 11:35:20.445235 2033 consensus_queue.cc:237] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } }
I20260926 11:35:20.445956 2057 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: TxnStatusTablet state changed. Reason: New leader 2108b87bbb26492782c29943d63cbf57. Latest consensus state: current_term: 1 leader_uuid: "2108b87bbb26492782c29943d63cbf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.446094 2057 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:35:20.446465 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 11:35:20.446633 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 11:35:20.447202 1508 catalog_manager.cc:5495] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2108b87bbb26492782c29943d63cbf57 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2108b87bbb26492782c29943d63cbf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:20.472340 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.468315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40451 (local address 127.0.0.1:42660)
0926 11:35:20.468463 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:20.468480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:20.468600 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:20.468910 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:20.468912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:20.468918 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:20.469012 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.469016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.470074 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.470078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.470794 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.470805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.471437 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.471440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.471524 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.471630 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:20.471994 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:20.472012 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 11:35:20.472083 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 11:35:20.472169 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 11:35:20.472324 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.468437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.0.98.1:40451)
0926 11:35:20.468567 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:20.468572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:20.468588 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:35:20.468740 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:20.468743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:20.468800 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:20.468890 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:20.469111 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.469931 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.470931 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.471277 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.471389 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.471782 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:20.471874 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:20.471950 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:20.472115 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 11:35:20.472177 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:35:20.475529 1805 raft_consensus.cc:1261] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531 [term 1 FOLLOWER]: Refusing update from remote peer 2108b87bbb26492782c29943d63cbf57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:20.475626 1945 raft_consensus.cc:1261] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Refusing update from remote peer 2108b87bbb26492782c29943d63cbf57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:20.476019 2057 consensus_queue.cc:1035] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:20.476179 2033 consensus_queue.cc:1035] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:20.477788 2022 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: TxnStatusTablet state changed. Reason: New leader 2108b87bbb26492782c29943d63cbf57. Latest consensus state: current_term: 1 leader_uuid: "2108b87bbb26492782c29943d63cbf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.477931 2022 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:20.477936 2020 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: TxnStatusTablet state changed. Reason: New leader 2108b87bbb26492782c29943d63cbf57. Latest consensus state: current_term: 1 leader_uuid: "2108b87bbb26492782c29943d63cbf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.478039 2020 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:20.480404 2033 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2108b87bbb26492782c29943d63cbf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.480523 2033 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:35:20.480643 2033 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2108b87bbb26492782c29943d63cbf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.480722 2033 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P 2108b87bbb26492782c29943d63cbf57: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:35:20.480891 2020 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2108b87bbb26492782c29943d63cbf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.480997 2020 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P d116ea0f6bba4e18ba432dc9e2564531: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:20.481889 2022 tablet_replica.cc:439] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2108b87bbb26492782c29943d63cbf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 } } peers { permanent_uuid: "d116ea0f6bba4e18ba432dc9e2564531" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46039 } } peers { permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 } } }
I20260926 11:35:20.482002 2022 tablet_replica.cc:442] T 07f20fcf4cca4a46bada745958682b34 P d0867a2a7055483f9a7d4f37e4614874: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:20.497159 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.493146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.0.98.2:46039)
0926 11:35:20.493414 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:20.493418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:20.493432 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:35:20.493608 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:20.493610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:20.493667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:20.493825 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:20.494155 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.494739 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.495613 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.495867 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.495983 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.496268 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:20.496740 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:20.496875 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:20.496939 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 11:35:20.496996 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":126,"threads_started":1}
I20260926 11:35:20.510226 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.506195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47413 (local address 127.0.98.1:40451)
0926 11:35:20.506337 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:20.506342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:20.506519 (+ 177us) server_negotiation.cc:407] Connection header received
0926 11:35:20.506621 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:20.506624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:20.506679 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:20.506762 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:20.506982 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.507804 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.508937 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.509274 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.509382 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.509652 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:20.509738 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:20.509808 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:20.510015 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 11:35:20.510080 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:35:20.510629 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.506082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40451 (local address 127.0.98.2:47413)
0926 11:35:20.506435 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:20.506449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:20.506543 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:20.506775 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:20.506777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:20.506782 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:20.506869 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.506873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.507931 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.507933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.508812 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.508826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.509409 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.509412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.509476 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.509522 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:20.509851 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:20.509859 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 11:35:20.510003 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 11:35:20.510101 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260926 11:35:20.519484 1945 raft_consensus.cc:1261] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Refusing update from remote peer d116ea0f6bba4e18ba432dc9e2564531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:20.519750 1665 raft_consensus.cc:1261] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [term 1 FOLLOWER]: Refusing update from remote peer d116ea0f6bba4e18ba432dc9e2564531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:20.520069 2077 consensus_queue.cc:1035] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0867a2a7055483f9a7d4f37e4614874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:20.520408 2046 consensus_queue.cc:1035] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2108b87bbb26492782c29943d63cbf57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:20.546468 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.542322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53705 (local address 127.0.98.2:46039)
0926 11:35:20.542456 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:20.542461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:20.542777 (+ 316us) server_negotiation.cc:407] Connection header received
0926 11:35:20.542902 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:20.542906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:20.543006 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:20.543098 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:20.543496 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.544170 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.545315 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.545558 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.545673 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.545989 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:20.546083 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:20.546169 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:20.546266 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 11:35:20.546323 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 11:35:20.546521 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.542273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46039 (local address 127.0.98.1:53705)
0926 11:35:20.542661 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:20.542679 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:20.542790 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:20.543228 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:20.543231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:20.543242 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:20.543388 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.543395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.544289 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.544295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.545133 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:20.545144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.545676 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.545679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:20.545776 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.545836 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:20.546186 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:20.546193 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:35:20.546255 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 11:35:20.546353 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
I20260926 11:35:20.577245 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:20.577463 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:20.580310 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1b02ccab9d8483a8585cb9e89cb9999: Sending DeleteTablet for 3 replicas of tablet f33ba5210936407bbb6e4f926da6d76e
I20260926 11:35:20.581404 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet f33ba5210936407bbb6e4f926da6d76e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56688
I20260926 11:35:20.581993 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet f33ba5210936407bbb6e4f926da6d76e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55782
I20260926 11:35:20.583062 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet f33ba5210936407bbb6e4f926da6d76e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260926 11:35:20.583495 2087 tablet_replica.cc:330] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: stopping tablet replica
I20260926 11:35:20.583709 2087 raft_consensus.cc:2229] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:20.584467 2087 raft_consensus.cc:2258] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:20.584793 2085 tablet_replica.cc:330] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: stopping tablet replica
I20260926 11:35:20.584962 2085 raft_consensus.cc:2229] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:20.585103 2085 raft_consensus.cc:2258] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:20.585392 2088 tablet_replica.cc:330] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: stopping tablet replica
I20260926 11:35:20.585541 2088 raft_consensus.cc:2229] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:20.585691 2088 raft_consensus.cc:2258] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:20.586571 2087 ts_tablet_manager.cc:1905] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:20.590426 2087 ts_tablet_manager.cc:1918] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:20.590553 2087 log.cc:1196] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/f33ba5210936407bbb6e4f926da6d76e
I20260926 11:35:20.590909 2087 ts_tablet_manager.cc:1939] T f33ba5210936407bbb6e4f926da6d76e P d116ea0f6bba4e18ba432dc9e2564531: Deleting consensus metadata
I20260926 11:35:20.591960 1493 catalog_manager.cc:4841] TS d116ea0f6bba4e18ba432dc9e2564531 (127.0.98.2:46039): tablet f33ba5210936407bbb6e4f926da6d76e (table test-table [id=492ab2ce044248468bf393bda0c53f90]) successfully deleted
I20260926 11:35:20.593319 2088 ts_tablet_manager.cc:1905] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 11:35:20 dist-test-slave-4pw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:35:20.594637 2085 ts_tablet_manager.cc:1905] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:20.598169 2085 ts_tablet_manager.cc:1918] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:20.598290 2085 log.cc:1196] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/f33ba5210936407bbb6e4f926da6d76e
I20260926 11:35:20.598623 2085 ts_tablet_manager.cc:1939] T f33ba5210936407bbb6e4f926da6d76e P d0867a2a7055483f9a7d4f37e4614874: Deleting consensus metadata
I20260926 11:35:20.599339 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.582968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.0.98.1:40451)
0926 11:35:20.583099 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:20.583104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:20.584280 (+ 1176us) server_negotiation.cc:407] Connection header received
0926 11:35:20.584510 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:20.584513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:20.584567 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:20.584627 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:20.586151 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.587151 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.588935 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.589281 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.589359 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.594368 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:20.594384 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:20.594404 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:20.594444 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:20.596836 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:20.597434 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:20.597441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:20.597444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:20.597539 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:20.597875 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:20.597879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:20.597881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:20.598209 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:20.598423 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:20.599005 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 11:35:20.599154 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:35:20.599659 1494 catalog_manager.cc:4841] TS d0867a2a7055483f9a7d4f37e4614874 (127.0.98.3:40943): tablet f33ba5210936407bbb6e4f926da6d76e (table test-table [id=492ab2ce044248468bf393bda0c53f90]) successfully deleted
W20260926 11:35:20.600841 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42676
I20260926 11:35:20.602998 2088 ts_tablet_manager.cc:1918] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:20.603108 2088 log.cc:1196] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/f33ba5210936407bbb6e4f926da6d76e
I20260926 11:35:20.603438 2088 ts_tablet_manager.cc:1939] T f33ba5210936407bbb6e4f926da6d76e P 2108b87bbb26492782c29943d63cbf57: Deleting consensus metadata
I20260926 11:35:20.604465 1494 catalog_manager.cc:4841] TS 2108b87bbb26492782c29943d63cbf57 (127.0.98.1:40451): tablet f33ba5210936407bbb6e4f926da6d76e (table test-table [id=492ab2ce044248468bf393bda0c53f90]) successfully deleted
I20260926 11:35:20.616052 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:20.603816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.0.98.62:36489)
0926 11:35:20.603989 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:20.603994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:20.604520 (+ 526us) server_negotiation.cc:407] Connection header received
0926 11:35:20.604816 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:20.604819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:20.604873 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:20.604955 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:20.606004 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.606924 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.608391 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:20.608818 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:20.608963 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:20.611187 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:20.611193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:20.611197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:20.611238 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:20.613705 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:20.614312 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:20.614317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:20.614319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:20.614443 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:20.614807 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:20.614811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:20.614814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:20.615104 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:20.615276 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:20.615676 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 11:35:20.615839 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 11:35:20.616678 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58260
I20260926 11:35:20.618561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 1580
I20260926 11:35:20.628238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 1720
I20260926 11:35:20.634635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 1860
I20260926 11:35:20.641217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 1476
2026-09-26T11:35:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3757 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:20 dist-test-slave-4pw5 krb5kdc[2094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:20 dist-test-slave-4pw5 krb5kdc[2094](info): set up 2 sockets
Sep 26 11:35:20 dist-test-slave-4pw5 krb5kdc[2094](info): commencing operation
krb5kdc: starting...
W20260926 11:35:21.760107 392 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:21 dist-test-slave-4pw5 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:21.907496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40935
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:22.025887 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:22.026225 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:22.026279 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:22.031229 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:22.031332 2107 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 11:35:22.031349 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:22.031368 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:22.031385 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:22.031401 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:22.037446 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40935
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:22.038703 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:22.039873 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:22.049333 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:22.049234 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:22.049232 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:22.049307 2107 server_base.cc:1034] running on GCE node
I20260926 11:35:22.050267 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:22.051541 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:22.052801 2107 hybrid_clock.cc:648] HybridClock initialized: now 1790422522052780 us; error 40 us; skew 500 ppm
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:22.056496 2107 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:22.057483 2107 webserver.cc:460] Webserver started at http://127.0.98.62:35219/ using document root <none> and password file <none>
I20260926 11:35:22.058060 2107 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:22.058125 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:22.058316 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:22.060585 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "230eee5d58e349db95a87a8e72078fac"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "0cd195ac20bc6c996a99c0a60de6e346"
server_key_iv: "41d7a721eeb7b326eee54f6e8f6e5b71"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.061134 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "230eee5d58e349db95a87a8e72078fac"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "0cd195ac20bc6c996a99c0a60de6e346"
server_key_iv: "41d7a721eeb7b326eee54f6e8f6e5b71"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.065487 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 11:35:22.068198 2122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:22.069581 2107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:22.069743 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "230eee5d58e349db95a87a8e72078fac"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "0cd195ac20bc6c996a99c0a60de6e346"
server_key_iv: "41d7a721eeb7b326eee54f6e8f6e5b71"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.069880 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:22.082881 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:22.086876 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:22.087137 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:22.096650 2107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40935
I20260926 11:35:22.096725 2184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40935 every 8 connection(s)
I20260926 11:35:22.097703 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:22.100906 2185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:22.105154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 2107
I20260926 11:35:22.105306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:22.105592 392 external_mini_cluster.cc:1428] Setting key 26fbbf860a9646b340b3ea8c27ccc96c
I20260926 11:35:22.107437 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: Bootstrap starting.
I20260926 11:35:22.110097 2185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:22.110881 2185 log.cc:824] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:22.112902 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: No bootstrap required, opened a new log
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:22.116598 2185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eee5d58e349db95a87a8e72078fac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40935 } }
I20260926 11:35:22.116917 2185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:22.116979 2185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230eee5d58e349db95a87a8e72078fac, State: Initialized, Role: FOLLOWER
I20260926 11:35:22.117482 2185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eee5d58e349db95a87a8e72078fac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40935 } }
I20260926 11:35:22.117609 2185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:22.117692 2185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:22.117803 2185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:22.118726 2185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eee5d58e349db95a87a8e72078fac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40935 } }
I20260926 11:35:22.119100 2185 leader_election.cc:302] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230eee5d58e349db95a87a8e72078fac; no voters:
I20260926 11:35:22.119422 2185 leader_election.cc:288] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:22.119546 2190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:22.119817 2190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [term 1 LEADER]: Becoming Leader. State: Replica: 230eee5d58e349db95a87a8e72078fac, State: Running, Role: LEADER
I20260926 11:35:22.120150 2190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eee5d58e349db95a87a8e72078fac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40935 } }
I20260926 11:35:22.120828 2185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:22.121371 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230eee5d58e349db95a87a8e72078fac. Latest consensus state: current_term: 1 leader_uuid: "230eee5d58e349db95a87a8e72078fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eee5d58e349db95a87a8e72078fac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40935 } } }
I20260926 11:35:22.121477 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:22.121788 2191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "230eee5d58e349db95a87a8e72078fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230eee5d58e349db95a87a8e72078fac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40935 } } }
I20260926 11:35:22.121904 2191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:22.121938 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.107091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.0.98.62:40935)
0926 11:35:22.107528 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.107545 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.107644 (+ 99us) server_negotiation.cc:407] Connection header received
0926 11:35:22.108307 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.108324 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.108658 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.108961 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:22.109898 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.111240 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.112305 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.112848 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.113041 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.115873 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:22.115879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:22.115896 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:22.115927 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:22.118102 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.118594 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.118607 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.118612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.118726 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.119023 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.119028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.119030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.119569 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:22.120418 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.120574 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.120886 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":150,"threads_started":1}
W20260926 11:35:22.125988 2205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:35:22.126081 2205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:35:22.126178 2195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:22.126932 2195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:22.132786 2195 catalog_manager.cc:1347] Generated new cluster ID: 9e6e56fddbe74077b44d992b392f8de6
I20260926 11:35:22.132920 2195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:22.161726 2195 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:22.163105 2195 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:22.170218 2195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: Generated new TSK 0
I20260926 11:35:22.171166 2195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:22.207866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40935
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:22.333597 2211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:22.333964 2211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:22.334059 2211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:22.338330 2211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:22.338495 2211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:22.338581 2211 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 11:35:22.338630 2211 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:35:22.338752 2211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:22.344667 2211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:22.345867 2211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:22.346951 2211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:22.357843 2217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:22.357931 2211 server_base.cc:1034] running on GCE node
W20260926 11:35:22.357935 2216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:22.357843 2219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:22.358659 2211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:22.359411 2211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:22.360637 2211 hybrid_clock.cc:648] HybridClock initialized: now 1790422522360613 us; error 49 us; skew 500 ppm
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:22.363894 2211 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:22.364951 2211 webserver.cc:460] Webserver started at http://127.0.98.1:39369/ using document root <none> and password file <none>
I20260926 11:35:22.365586 2211 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:22.365674 2211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:22.365898 2211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:22.368163 2211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1839855de2c4fa0ab1232699126d88a"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "74fd2d669bdf905d1debb53a50699a6a"
server_key_iv: "95828cb030a93fa981d4cd7c063547c9"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.368738 2211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1839855de2c4fa0ab1232699126d88a"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "74fd2d669bdf905d1debb53a50699a6a"
server_key_iv: "95828cb030a93fa981d4cd7c063547c9"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.372677 2211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:22.375490 2226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:22.377018 2211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:22.377198 2211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "e1839855de2c4fa0ab1232699126d88a"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "74fd2d669bdf905d1debb53a50699a6a"
server_key_iv: "95828cb030a93fa981d4cd7c063547c9"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.377343 2211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:22.392282 2211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:22.396364 2211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:22.396605 2211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:22.398353 2211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:22.398437 2211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:22.398515 2211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:22.398564 2211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:22.410964 2211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40627
I20260926 11:35:22.410991 2341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40627 every 8 connection(s)
I20260926 11:35:22.412099 2211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:22.414633 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.399340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53133 (local address 127.0.98.62:40935)
0926 11:35:22.399525 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.399533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.400665 (+ 1132us) server_negotiation.cc:407] Connection header received
0926 11:35:22.402143 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.402149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.402219 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.402356 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:22.403785 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.404813 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.406278 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.406765 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.406909 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.410472 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:22.410477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:22.410481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:22.410520 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:22.412393 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.413016 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.413020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.413024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.413112 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.413471 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.413474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.413476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.413694 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:22.413804 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.414325 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.414464 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 11:35:22.415524 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.399898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40935 (local address 127.0.98.1:53133)
0926 11:35:22.400455 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:22.400522 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:22.401797 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:22.402581 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:22.402588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:22.402952 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:22.403563 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.403606 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.404985 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.404989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.406134 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.406144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.406984 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.406990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.407244 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.408631 (+ 1387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:22.408650 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:22.410239 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:22.412530 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.412536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.412551 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.412901 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.413208 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.413213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.413215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.413363 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.413802 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:22.413809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:22.414073 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 11:35:22.414266 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 11:35:22.414626 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":127,"threads_started":1}
I20260926 11:35:22.415897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 2211
I20260926 11:35:22.416059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:22.416330 392 external_mini_cluster.cc:1428] Setting key 5ed7074cb1f5ba7737c19f107a43b040
I20260926 11:35:22.417138 2342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40935
I20260926 11:35:22.417323 2342 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:22.417975 2342 heartbeater.cc:511] Master 127.0.98.62:40935 requested a full tablet report, sending...
I20260926 11:35:22.419620 2138 ts_manager.cc:194] Registered new tserver with Master: e1839855de2c4fa0ab1232699126d88a (127.0.98.1:40627)
I20260926 11:35:22.420917 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53133
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:35:22.428988 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.419707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60299 (local address 127.0.98.62:40935)
0926 11:35:22.419850 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.419857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.421291 (+ 1434us) server_negotiation.cc:407] Connection header received
0926 11:35:22.421428 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.421432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.421525 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.421602 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:22.422219 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.422869 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.423789 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.424025 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.424136 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.425319 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:22.425323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:22.425326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:22.425353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:22.427123 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.427632 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.427635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.427637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.427732 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.428094 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.428097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.428098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.428290 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:22.428387 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.428704 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.428841 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:35:22.429023 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.419627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40935 (local address 127.0.98.1:60299)
0926 11:35:22.421165 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:22.421177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:22.421320 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:22.421715 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:22.421718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:22.421995 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:22.422098 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.422104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.422996 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.422999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.423673 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.423683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.424140 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.424143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.424235 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.424800 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:22.424813 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:22.425174 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:22.427247 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.427251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.427253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.427496 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.427842 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.427851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.427853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.427958 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.428415 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:22.428418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:22.428516 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 11:35:22.428631 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 11:35:22.428790 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1458,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:35:22.447216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40935
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:22.561790 2351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:22.562099 2351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:22.562150 2351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:22.567116 2351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:22.567207 2351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:22.567253 2351 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 11:35:22.567271 2351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:35:22.567305 2351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:22.573236 2351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:22.574666 2351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:22.575945 2351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:22.587522 2356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:22.587527 2357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:22.587522 2359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:22.587972 2351 server_base.cc:1034] running on GCE node
I20260926 11:35:22.588582 2351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:22.589253 2351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:22.590513 2351 hybrid_clock.cc:648] HybridClock initialized: now 1790422522590491 us; error 39 us; skew 500 ppm
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:22.593950 2351 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:35:22.595204 2351 webserver.cc:460] Webserver started at http://127.0.98.2:40947/ using document root <none> and password file <none>
I20260926 11:35:22.595969 2351 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:22.596076 2351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:22.596359 2351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:22.598798 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "89132af77eb644c49f88e7198c025b5a"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "168c1dc1887976a78d649ff78240f7d5"
server_key_iv: "b2e65d6709fd50aa0b6fc45bdfcd28d2"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.599635 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "89132af77eb644c49f88e7198c025b5a"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "168c1dc1887976a78d649ff78240f7d5"
server_key_iv: "b2e65d6709fd50aa0b6fc45bdfcd28d2"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.604084 2351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 11:35:22.606974 2366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:22.608597 2351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:22.608780 2351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "89132af77eb644c49f88e7198c025b5a"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "168c1dc1887976a78d649ff78240f7d5"
server_key_iv: "b2e65d6709fd50aa0b6fc45bdfcd28d2"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.608932 2351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:22.637660 2351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:22.641453 2351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:22.641692 2351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:22.643846 2351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:22.643930 2351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:22.644009 2351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:22.644044 2351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:22.656584 2351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35967
I20260926 11:35:22.656675 2481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35967 every 8 connection(s)
I20260926 11:35:22.657639 2351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:22.659384 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.644683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51339 (local address 127.0.98.62:40935)
0926 11:35:22.644856 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.644861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.645831 (+ 970us) server_negotiation.cc:407] Connection header received
0926 11:35:22.647249 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.647270 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.647412 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.647485 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:22.649657 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.650352 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.651698 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.652076 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.652205 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.655329 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:22.655335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:22.655337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:22.655372 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:22.657125 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.657693 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.657696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.657698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.657819 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.658233 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.658236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.658238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.658503 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:22.658609 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.659144 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.659260 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:35:22.660352 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.645138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40935 (local address 127.0.98.2:51339)
0926 11:35:22.645652 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:22.645714 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:22.646948 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:22.648028 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:22.648039 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:22.648601 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:22.649445 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.649471 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.650522 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.650526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.651491 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.651503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.652275 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.652279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.652487 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.653238 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:22.653257 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:22.655120 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:22.657270 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.657276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.657292 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.657573 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.657924 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.657927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.657929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.658090 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.658629 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:22.658635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:22.658898 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 11:35:22.659088 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 11:35:22.659396 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260926 11:35:22.661904 2482 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40935
I20260926 11:35:22.662125 2482 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:22.662765 2482 heartbeater.cc:511] Master 127.0.98.62:40935 requested a full tablet report, sending...
I20260926 11:35:22.664108 2138 ts_manager.cc:194] Registered new tserver with Master: 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967)
I20260926 11:35:22.665050 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51339
I20260926 11:35:22.665400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 2351
I20260926 11:35:22.665514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:22.665802 392 external_mini_cluster.cc:1428] Setting key 3ca637eba2535c8da74eb5dda86addff
I20260926 11:35:22.672977 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.664121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40935 (local address 127.0.98.2:55679)
0926 11:35:22.664463 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:22.664476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:22.664564 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:22.664818 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:22.664820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:22.665021 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:22.665180 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.665189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.666186 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.666191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.667144 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.667157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.667703 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.667707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.667823 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.668447 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:22.668460 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:22.668887 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:22.670973 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.670977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.670979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.671212 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.671692 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.671695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.671698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.671815 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.672448 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:22.672451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:22.672545 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 11:35:22.672654 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 11:35:22.672807 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":96,"threads_started":1}
I20260926 11:35:22.673148 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.664264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55679 (local address 127.0.98.62:40935)
0926 11:35:22.664444 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.664449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.664552 (+ 103us) server_negotiation.cc:407] Connection header received
0926 11:35:22.664633 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.664637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.664684 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.664800 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:22.665317 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.666038 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.667262 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.667523 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.667698 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.669052 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:22.669056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:22.669059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:22.669091 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:22.670844 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.671369 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.671376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.671380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.671488 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.671929 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.671935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.671940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.672326 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:22.672454 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.672742 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.672889 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:35:22.696543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40935
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:22.816763 2491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:22.817073 2491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:22.817122 2491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:22.822127 2491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:22.822211 2491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:22.822258 2491 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 11:35:22.822274 2491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:35:22.822309 2491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:22.828099 2491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:22.829233 2491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:22.830271 2491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:22.840466 2499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:22.840466 2497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:22.840476 2496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:22.840812 2491 server_base.cc:1034] running on GCE node
I20260926 11:35:22.841250 2491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:22.841935 2491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:22.843146 2491 hybrid_clock.cc:648] HybridClock initialized: now 1790422522843107 us; error 52 us; skew 500 ppm
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:22.846387 2491 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:35:22.847399 2491 webserver.cc:460] Webserver started at http://127.0.98.3:36559/ using document root <none> and password file <none>
I20260926 11:35:22.848030 2491 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:22.848097 2491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:22.848278 2491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:22.850419 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "8c70782a6e9641f998a1c3f88baaad0e"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "8d2908db542ab795d6a8eea6ffc6de0f"
server_key_iv: "cb09f3e224743faffdc090a6793f86e5"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.850991 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8c70782a6e9641f998a1c3f88baaad0e"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "8d2908db542ab795d6a8eea6ffc6de0f"
server_key_iv: "cb09f3e224743faffdc090a6793f86e5"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.855116 2491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260926 11:35:22.858042 2506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:22.859840 2491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:22.859992 2491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "8c70782a6e9641f998a1c3f88baaad0e"
format_stamp: "Formatted at 2026-09-26 11:35:22 on dist-test-slave-4pw5"
server_key: "8d2908db542ab795d6a8eea6ffc6de0f"
server_key_iv: "cb09f3e224743faffdc090a6793f86e5"
server_key_version: "encryptionkey@0"
I20260926 11:35:22.860122 2491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:22.873935 2491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:22.877514 2491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:22.877748 2491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:22.879422 2491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:22.879477 2491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:22.879518 2491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:22.879535 2491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:22.891211 2491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40229
I20260926 11:35:22.891258 2621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40229 every 8 connection(s)
I20260926 11:35:22.892454 2491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:22.894327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 2491
I20260926 11:35:22.894425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:22.894740 392 external_mini_cluster.cc:1428] Setting key a70322f17e009dbffc82c48cd5ecf425
I20260926 11:35:22.895886 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.880188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38863 (local address 127.0.98.62:40935)
0926 11:35:22.880353 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.880358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.881421 (+ 1063us) server_negotiation.cc:407] Connection header received
0926 11:35:22.882901 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.882908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.882984 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.883051 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:22.885161 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.885728 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.886956 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.887261 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.887379 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.890265 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:22.890270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:22.890273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:22.890305 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:22.892224 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.893951 (+ 1727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.893954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.893955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.894098 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.894568 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.894570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.894572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.894831 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:22.894933 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.895557 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.895725 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:35:22.896860 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.880658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40935 (local address 127.0.98.3:38863)
0926 11:35:22.881234 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:22.881296 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:22.882600 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:22.883540 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:22.883552 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:22.884167 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:22.884978 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.885004 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.885877 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.885881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.886813 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.886824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.887447 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.887451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.887698 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.888436 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:22.888453 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:22.890041 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:22.892361 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.892368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.892384 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.892777 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.894219 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.894223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.894225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.894440 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.894944 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:22.894953 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:22.895265 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 11:35:22.895491 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 11:35:22.895927 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":177,"threads_started":1}
I20260926 11:35:22.898191 2622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40935
I20260926 11:35:22.898324 2622 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:22.898895 2622 heartbeater.cc:511] Master 127.0.98.62:40935 requested a full tablet report, sending...
I20260926 11:35:22.900019 2138 ts_manager.cc:194] Registered new tserver with Master: 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229)
I20260926 11:35:22.900635 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38863
I20260926 11:35:22.908813 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.901250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54851 (local address 127.0.98.62:40935)
0926 11:35:22.901427 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.901430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.901606 (+ 176us) server_negotiation.cc:407] Connection header received
0926 11:35:22.901737 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.901741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.901800 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.901899 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:22.902452 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.903020 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.904012 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.904255 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.904381 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.905574 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:22.905577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:22.905579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:22.905607 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:22.907116 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.907568 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.907607 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.907609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.907690 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.907972 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.907977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.907979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.908157 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:22.908264 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.908518 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.908630 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 11:35:22.908819 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.901122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40935 (local address 127.0.98.3:54851)
0926 11:35:22.901500 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:22.901514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:22.901648 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:22.901916 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:22.901920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:22.902161 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:22.902264 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.902270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.903162 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.903165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.903871 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.903881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.904383 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.904387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.904493 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.904997 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:22.905009 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:22.905404 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:22.907241 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.907246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.907248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.907464 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.907795 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:22.907798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:22.907800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:22.907880 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:22.908253 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:22.908256 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:22.908358 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 11:35:22.908464 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 11:35:22.908632 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":114,"threads_started":1}
I20260926 11:35:22.908991 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 11:35:22 dist-test-slave-4pw5 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:22.938246 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.929002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.0.98.62:40935)
0926 11:35:22.929134 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.929138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.929151 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:35:22.929195 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.929197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.929236 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.929316 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:22.930097 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.930668 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.931829 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.932046 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.932127 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.935004 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:22.935008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:22.935010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:22.935037 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:22.936645 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.937098 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.937101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.937102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.937175 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:22.937452 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:22.937454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:22.937455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:22.937633 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:22.937728 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.938027 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.938122 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:35:22.941645 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:22.944764 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:22.962035 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.954757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40229 (local address 127.0.0.1:37566)
0926 11:35:22.955200 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:22.955257 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:22.955439 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:22.956309 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:22.956317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:22.956346 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:22.956546 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.956555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.958415 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.958422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.960390 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.960401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.961506 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.961509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.961601 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.961623 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:35:22.961672 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 11:35:22.961848 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":472,"thread_start_us":100,"threads_started":1}
I20260926 11:35:22.962423 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.955709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.0.98.2:35967)
0926 11:35:22.956613 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.956626 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.956651 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:35:22.956726 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.956730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.956899 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.957123 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:22.957451 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.958640 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.960427 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.961138 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.961328 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.961584 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.961726 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.961854 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":103,"threads_started":1}
I20260926 11:35:22.962688 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.955014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35967 (local address 127.0.0.1:35228)
0926 11:35:22.956139 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:22.956153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:22.956257 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:22.957141 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:22.957145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:22.957165 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:22.957289 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.957296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.958831 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.958838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.960278 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.960289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.961350 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.961355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.961489 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.961542 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 11:35:22.961693 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 11:35:22.961853 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":271,"thread_start_us":122,"threads_started":1}
I20260926 11:35:22.963338 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.954933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.0.98.3:40229)
0926 11:35:22.955668 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.955688 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.955720 (+ 32us) server_negotiation.cc:407] Connection header received
0926 11:35:22.955827 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.955834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.956088 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.956760 (+ 672us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:22.956942 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.958277 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.960538 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.961355 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.961945 (+ 590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.962108 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.963071 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.963180 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":144,"threads_started":1}
I20260926 11:35:22.963725 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.956326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.0.98.1:40627)
0926 11:35:22.956636 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:22.956643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:22.956911 (+ 268us) server_negotiation.cc:407] Connection header received
0926 11:35:22.957062 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:22.957068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:22.957240 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:22.957398 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:22.957616 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.958336 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.960538 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.961435 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.961529 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.961673 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:22.962112 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 11:35:22.962239 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":101,"threads_started":1}
I20260926 11:35:22.964010 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:22.955850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40627 (local address 127.0.0.1:54610)
0926 11:35:22.956784 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:22.956801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:22.956921 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:22.957392 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:22.957395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:22.957411 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:22.957500 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.957506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.958488 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.958491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.960390 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:22.960401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:22.961729 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:22.961732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:22.961813 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:22.961834 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:35:22.961902 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 11:35:22.961966 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":402,"thread_start_us":103,"threads_started":1}
I20260926 11:35:22.966167 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5abd0248764f1997634f15a75fbc19 (DEFAULT_TABLE table=test-table [id=88509bfc06c94c5cb990c0fc8dfb56a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:22.966269 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5abd0248764f1997634f15a75fbc19 (DEFAULT_TABLE table=test-table [id=88509bfc06c94c5cb990c0fc8dfb56a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:22.966812 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5abd0248764f1997634f15a75fbc19 (DEFAULT_TABLE table=test-table [id=88509bfc06c94c5cb990c0fc8dfb56a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:22.967238 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5abd0248764f1997634f15a75fbc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:22.967429 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5abd0248764f1997634f15a75fbc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:22.967873 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5abd0248764f1997634f15a75fbc19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:22.974097 2645 tablet_bootstrap.cc:492] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: Bootstrap starting.
I20260926 11:35:22.975454 2646 tablet_bootstrap.cc:492] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: Bootstrap starting.
I20260926 11:35:22.976794 2645 tablet_bootstrap.cc:654] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:22.977614 2646 tablet_bootstrap.cc:654] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:22.978147 2645 log.cc:824] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:22.978443 2646 log.cc:824] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:22.978610 2647 tablet_bootstrap.cc:492] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: Bootstrap starting.
I20260926 11:35:22.980391 2646 tablet_bootstrap.cc:492] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: No bootstrap required, opened a new log
I20260926 11:35:22.980378 2645 tablet_bootstrap.cc:492] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: No bootstrap required, opened a new log
I20260926 11:35:22.980590 2646 ts_tablet_manager.cc:1397] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:35:22.980613 2645 ts_tablet_manager.cc:1397] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 11:35:22.980749 2647 tablet_bootstrap.cc:654] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:22.981745 2647 log.cc:824] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:22.984237 2647 tablet_bootstrap.cc:492] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: No bootstrap required, opened a new log
I20260926 11:35:22.984450 2647 ts_tablet_manager.cc:1397] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:35:22.985105 2645 raft_consensus.cc:348] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } }
I20260926 11:35:22.985110 2646 raft_consensus.cc:348] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } }
I20260926 11:35:22.985397 2646 raft_consensus.cc:374] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:22.985399 2645 raft_consensus.cc:374] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:22.985461 2645 raft_consensus.cc:729] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c70782a6e9641f998a1c3f88baaad0e, State: Initialized, Role: FOLLOWER
I20260926 11:35:22.985469 2646 raft_consensus.cc:729] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1839855de2c4fa0ab1232699126d88a, State: Initialized, Role: FOLLOWER
I20260926 11:35:22.985951 2645 consensus_queue.cc:260] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } }
I20260926 11:35:22.985957 2646 consensus_queue.cc:260] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } }
I20260926 11:35:22.986708 2342 heartbeater.cc:503] Master 127.0.98.62:40935 was elected leader, sending a full tablet report...
I20260926 11:35:22.987031 2646 ts_tablet_manager.cc:1428] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:35:22.987044 2645 ts_tablet_manager.cc:1428] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 11:35:22.987812 2622 heartbeater.cc:503] Master 127.0.98.62:40935 was elected leader, sending a full tablet report...
I20260926 11:35:22.988408 2647 raft_consensus.cc:348] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } }
I20260926 11:35:22.988615 2647 raft_consensus.cc:374] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:22.988654 2647 raft_consensus.cc:729] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89132af77eb644c49f88e7198c025b5a, State: Initialized, Role: FOLLOWER
I20260926 11:35:22.989073 2647 consensus_queue.cc:260] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } }
I20260926 11:35:22.989809 2482 heartbeater.cc:503] Master 127.0.98.62:40935 was elected leader, sending a full tablet report...
I20260926 11:35:22.990113 2647 ts_tablet_manager.cc:1428] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:35:23.106736 2651 raft_consensus.cc:482] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:23.107053 2651 raft_consensus.cc:504] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } }
I20260926 11:35:23.108633 2651 leader_election.cc:288] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229), 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967)
I20260926 11:35:23.113111 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.108925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35967 (local address 127.0.98.1:43899)
0926 11:35:23.109435 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:23.109450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:23.109550 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:23.109870 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:23.109873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:23.109886 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:23.109980 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.109985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.110917 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.110919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.111883 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.111892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.112623 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.112627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.112741 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.112792 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 11:35:23.112868 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 11:35:23.112924 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":84,"thread_start_us":103,"threads_started":1}
I20260926 11:35:23.113242 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.109295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33111 (local address 127.0.98.3:40229)
0926 11:35:23.109438 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:23.109444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:23.109463 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:35:23.109599 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:23.109604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:23.109657 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:23.109724 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:23.110178 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.110682 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.112058 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.112477 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.112593 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.112660 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:23.113040 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 11:35:23.113119 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:35:23.113394 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.108826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40229 (local address 127.0.98.1:33111)
0926 11:35:23.109321 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:23.109340 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:23.109473 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:23.109789 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:23.109799 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:23.109823 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:23.110003 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.110010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.110812 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.110818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.111896 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.111907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.112617 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.112621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.112892 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.112924 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 11:35:23.113009 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:35:23.113072 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":174,"thread_start_us":128,"threads_started":1}
I20260926 11:35:23.113149 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.109069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43899 (local address 127.0.98.2:35967)
0926 11:35:23.109560 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:23.109566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:23.109585 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:35:23.109654 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:23.109657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:23.109730 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:23.109829 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:23.110105 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.110713 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.112010 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.112477 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.112626 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.112698 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:23.112925 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 11:35:23.113024 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:35:23.114228 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5abd0248764f1997634f15a75fbc19" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c70782a6e9641f998a1c3f88baaad0e" is_pre_election: true
I20260926 11:35:23.114323 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5abd0248764f1997634f15a75fbc19" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a" is_pre_election: true
I20260926 11:35:23.114588 2576 raft_consensus.cc:2454] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 0.
I20260926 11:35:23.114612 2436 raft_consensus.cc:2454] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 0.
I20260926 11:35:23.115077 2227 leader_election.cc:302] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 89132af77eb644c49f88e7198c025b5a, 8c70782a6e9641f998a1c3f88baaad0e, e1839855de2c4fa0ab1232699126d88a; no voters:
I20260926 11:35:23.115348 2651 raft_consensus.cc:2781] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:23.115444 2651 raft_consensus.cc:482] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:23.115500 2651 raft_consensus.cc:3037] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:23.116565 2651 raft_consensus.cc:504] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } }
I20260926 11:35:23.116940 2651 leader_election.cc:288] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 election: Requested vote from peers 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229), 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967)
I20260926 11:35:23.117365 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5abd0248764f1997634f15a75fbc19" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c70782a6e9641f998a1c3f88baaad0e"
I20260926 11:35:23.117342 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5abd0248764f1997634f15a75fbc19" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a"
I20260926 11:35:23.117494 2436 raft_consensus.cc:3037] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:23.117504 2576 raft_consensus.cc:3037] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:23.118708 2436 raft_consensus.cc:2454] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 1.
I20260926 11:35:23.118861 2576 raft_consensus.cc:2454] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 1.
I20260926 11:35:23.119050 2227 leader_election.cc:302] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89132af77eb644c49f88e7198c025b5a, e1839855de2c4fa0ab1232699126d88a; no voters:
I20260926 11:35:23.119222 2651 raft_consensus.cc:2781] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:23.119450 2651 raft_consensus.cc:686] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 1 LEADER]: Becoming Leader. State: Replica: e1839855de2c4fa0ab1232699126d88a, State: Running, Role: LEADER
I20260926 11:35:23.119827 2651 consensus_queue.cc:237] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } }
I20260926 11:35:23.123387 2139 catalog_manager.cc:5495] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1839855de2c4fa0ab1232699126d88a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } health_report { overall_health: HEALTHY } } }
W20260926 11:35:23.143644 2623 tablet.cc:2369] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:23.158950 2483 tablet.cc:2369] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:23.163726 2343 tablet.cc:2369] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 11:35:23 dist-test-slave-4pw5 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:23.196580 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.185517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40935 (local address 127.0.0.1:41554)
0926 11:35:23.185829 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:23.185858 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:23.185962 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:23.186273 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:23.186277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:23.186749 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:23.186870 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.186878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.187868 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.187871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.188498 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.188507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.188983 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.188987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.189073 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.189717 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:23.189729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:23.191418 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:23.193848 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:23.193859 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:23.193866 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:23.194314 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:23.194658 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:23.194662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:23.194664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:23.194866 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:23.195515 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:23.195524 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:23.195777 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 11:35:23.195969 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 11:35:23.196233 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":126,"threads_started":1}
I20260926 11:35:23.196580 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.185669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.0.98.62:40935)
0926 11:35:23.185817 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:23.185823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:23.185934 (+ 111us) server_negotiation.cc:407] Connection header received
0926 11:35:23.186052 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:23.186054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:23.186097 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:23.186188 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:23.187018 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.187707 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.188625 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.188851 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.188942 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.191667 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:23.191671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:23.191673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:23.191696 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:23.193676 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:23.194435 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:23.194441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:23.194445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:23.194543 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:23.195065 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:23.195069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:23.195070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:23.195357 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:23.195478 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:23.196050 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 11:35:23.196336 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":52}
I20260926 11:35:23.202795 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41554:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 11:35:23.203559 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:23.209124 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 4e603fc0984a465b884f8a561bc9ab07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c468b01d8ee49698c1bd6140c84ba44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:35:23.209170 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 4e603fc0984a465b884f8a561bc9ab07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c468b01d8ee49698c1bd6140c84ba44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:35:23.209920 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e603fc0984a465b884f8a561bc9ab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:23.209925 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e603fc0984a465b884f8a561bc9ab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:23.212906 2645 tablet_bootstrap.cc:492] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: Bootstrap starting.
I20260926 11:35:23.213229 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 4e603fc0984a465b884f8a561bc9ab07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c468b01d8ee49698c1bd6140c84ba44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:35:23.213923 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e603fc0984a465b884f8a561bc9ab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:23.213939 2645 tablet_bootstrap.cc:654] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:23.215790 2647 tablet_bootstrap.cc:492] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: Bootstrap starting.
I20260926 11:35:23.216656 2645 tablet_bootstrap.cc:492] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: No bootstrap required, opened a new log
I20260926 11:35:23.216758 2645 ts_tablet_manager.cc:1397] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260926 11:35:23.216765 2647 tablet_bootstrap.cc:654] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:23.217213 2646 tablet_bootstrap.cc:492] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: Bootstrap starting.
I20260926 11:35:23.217263 2645 raft_consensus.cc:348] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.217406 2645 raft_consensus.cc:374] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:23.217453 2645 raft_consensus.cc:729] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c70782a6e9641f998a1c3f88baaad0e, State: Initialized, Role: FOLLOWER
I20260926 11:35:23.217664 2645 consensus_queue.cc:260] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.217986 2645 ts_tablet_manager.cc:1428] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:23.217952 2652 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.218084 2652 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:23.218142 2647 tablet_bootstrap.cc:492] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: No bootstrap required, opened a new log
I20260926 11:35:23.218220 2647 ts_tablet_manager.cc:1397] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:35:23.218648 2646 tablet_bootstrap.cc:654] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:23.219260 2647 raft_consensus.cc:348] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.219398 2647 raft_consensus.cc:374] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:23.219426 2647 raft_consensus.cc:729] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89132af77eb644c49f88e7198c025b5a, State: Initialized, Role: FOLLOWER
I20260926 11:35:23.219633 2647 consensus_queue.cc:260] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.219995 2653 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.220110 2646 tablet_bootstrap.cc:492] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: No bootstrap required, opened a new log
I20260926 11:35:23.220148 2653 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:23.220189 2646 ts_tablet_manager.cc:1397] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:23.220384 2647 ts_tablet_manager.cc:1428] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:35:23.220781 2646 raft_consensus.cc:348] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.220942 2646 raft_consensus.cc:374] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:23.220986 2646 raft_consensus.cc:729] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1839855de2c4fa0ab1232699126d88a, State: Initialized, Role: FOLLOWER
I20260926 11:35:23.221115 2646 consensus_queue.cc:260] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.221437 2646 ts_tablet_manager.cc:1428] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 11:35:23.221412 2651 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.221530 2651 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:23.265532 2651 raft_consensus.cc:482] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:23.265689 2651 raft_consensus.cc:504] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.266095 2651 leader_election.cc:288] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967), 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229)
I20260926 11:35:23.266691 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e603fc0984a465b884f8a561bc9ab07" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a" is_pre_election: true
I20260926 11:35:23.266691 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e603fc0984a465b884f8a561bc9ab07" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c70782a6e9641f998a1c3f88baaad0e" is_pre_election: true
I20260926 11:35:23.266865 2576 raft_consensus.cc:2454] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 0.
I20260926 11:35:23.266865 2436 raft_consensus.cc:2454] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 0.
I20260926 11:35:23.267252 2227 leader_election.cc:302] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89132af77eb644c49f88e7198c025b5a, e1839855de2c4fa0ab1232699126d88a; no voters:
I20260926 11:35:23.267503 2651 raft_consensus.cc:2781] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:23.267606 2651 raft_consensus.cc:482] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:23.267657 2651 raft_consensus.cc:3037] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:23.268667 2651 raft_consensus.cc:504] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.269062 2651 leader_election.cc:288] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 election: Requested vote from peers 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967), 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229)
I20260926 11:35:23.269495 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e603fc0984a465b884f8a561bc9ab07" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c70782a6e9641f998a1c3f88baaad0e"
I20260926 11:35:23.269495 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e603fc0984a465b884f8a561bc9ab07" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a"
I20260926 11:35:23.269629 2576 raft_consensus.cc:3037] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:23.269629 2436 raft_consensus.cc:3037] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:23.270776 2436 raft_consensus.cc:2454] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 1.
I20260926 11:35:23.270776 2576 raft_consensus.cc:2454] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 1.
I20260926 11:35:23.271160 2227 leader_election.cc:302] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89132af77eb644c49f88e7198c025b5a, e1839855de2c4fa0ab1232699126d88a; no voters:
I20260926 11:35:23.271615 2651 raft_consensus.cc:2781] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:23.271795 2651 raft_consensus.cc:686] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [term 1 LEADER]: Becoming Leader. State: Replica: e1839855de2c4fa0ab1232699126d88a, State: Running, Role: LEADER
I20260926 11:35:23.271984 2651 consensus_queue.cc:237] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.273057 2656 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: TxnStatusTablet state changed. Reason: New leader e1839855de2c4fa0ab1232699126d88a. Latest consensus state: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.273198 2656 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:35:23.273579 2668 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 11:35:23.273865 2668 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 11:35:23.274410 2139 catalog_manager.cc:5495] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1839855de2c4fa0ab1232699126d88a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:23.303297 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.298224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.0.98.1:40627)
0926 11:35:23.298448 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:23.298454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:23.298472 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:35:23.298657 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:23.298660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:23.298733 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:23.298869 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:23.299899 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.300499 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.301638 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.301873 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.301997 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.302304 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:23.302817 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:23.302954 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:23.303073 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 11:35:23.303140 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:35:23.303297 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.298073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40627 (local address 127.0.0.1:54618)
0926 11:35:23.298299 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:23.298320 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:23.298472 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:23.298920 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:23.298923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:23.298932 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:23.299716 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.299734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.300632 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.300637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.301429 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.301439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.302003 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.302006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.302097 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.302149 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:23.302923 (+ 774us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:23.302945 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:35:23.303050 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 11:35:23.303153 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 11:35:23.309713 2436 raft_consensus.cc:1261] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Refusing update from remote peer e1839855de2c4fa0ab1232699126d88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:23.309719 2576 raft_consensus.cc:1261] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Refusing update from remote peer e1839855de2c4fa0ab1232699126d88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:23.310616 2656 consensus_queue.cc:1035] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:23.310917 2651 consensus_queue.cc:1035] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:23.315265 2652 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: TxnStatusTablet state changed. Reason: New leader e1839855de2c4fa0ab1232699126d88a. Latest consensus state: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.315438 2652 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:23.315771 2653 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: TxnStatusTablet state changed. Reason: New leader e1839855de2c4fa0ab1232699126d88a. Latest consensus state: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.315917 2653 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:23.320225 2656 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.320410 2656 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:35:23.320734 2652 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.320863 2652 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P 8c70782a6e9641f998a1c3f88baaad0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:23.321208 2671 mvcc.cc:205] Tried to move back new op lower bound from 7333570655468699648 to 7333570655323672576. Current Snapshot: MvccSnapshot[applied={T|T < 7333570655468699648}]
I20260926 11:35:23.322841 2656 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.322958 2656 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P e1839855de2c4fa0ab1232699126d88a: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:35:23.323887 2673 mvcc.cc:205] Tried to move back new op lower bound from 7333570655468699648 to 7333570655323672576. Current Snapshot: MvccSnapshot[applied={T|T < 7333570655468699648}]
I20260926 11:35:23.325584 2653 tablet_replica.cc:439] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } } }
I20260926 11:35:23.325736 2653 tablet_replica.cc:442] T 4e603fc0984a465b884f8a561bc9ab07 P 89132af77eb644c49f88e7198c025b5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:23.326783 2672 mvcc.cc:205] Tried to move back new op lower bound from 7333570655468699648 to 7333570655323672576. Current Snapshot: MvccSnapshot[applied={T|T < 7333570655468699648}]
I20260926 11:35:23.333105 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.328624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.0.98.1:40627)
0926 11:35:23.328749 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:23.328754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:23.328930 (+ 176us) server_negotiation.cc:407] Connection header received
0926 11:35:23.329038 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:23.329041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:23.329093 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:23.329167 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:23.329548 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.330344 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.331706 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.331982 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.332086 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.332442 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:23.332538 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:23.332617 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:23.332745 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 11:35:23.332810 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 11:35:23.347306 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.342923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57781 (local address 127.0.98.1:40627)
0926 11:35:23.343071 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:23.343076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:23.343359 (+ 283us) server_negotiation.cc:407] Connection header received
0926 11:35:23.343503 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:23.343506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:23.343558 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:23.343677 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:23.343935 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.344743 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.345741 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.346092 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.346199 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.346749 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:23.346842 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:23.346915 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:23.347078 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 11:35:23.347138 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:35:23.347314 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:23.342810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40627 (local address 127.0.98.1:57781)
0926 11:35:23.343253 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:23.343269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:23.343398 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:23.343692 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:23.343695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:23.343701 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:23.343824 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.343829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.344889 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.344891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.345613 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:23.345622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:23.346271 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:23.346274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:23.346504 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:23.346562 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:23.346961 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:23.346968 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:35:23.347053 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:35:23.347151 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":1,"thread_start_us":173,"threads_started":1}
I20260926 11:35:23.355206 2436 raft_consensus.cc:1261] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Refusing update from remote peer e1839855de2c4fa0ab1232699126d88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:23.355355 2576 raft_consensus.cc:1261] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Refusing update from remote peer e1839855de2c4fa0ab1232699126d88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:23.355875 2678 consensus_queue.cc:1035] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:23.356130 2651 consensus_queue.cc:1035] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:23.409874 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:23.410101 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:23.413883 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: Sending DeleteTablet for 3 replicas of tablet 6c5abd0248764f1997634f15a75fbc19
I20260926 11:35:23.414865 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5abd0248764f1997634f15a75fbc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37566
I20260926 11:35:23.415441 2687 tablet_replica.cc:330] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: stopping tablet replica
I20260926 11:35:23.415880 2687 raft_consensus.cc:2229] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:23.415947 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5abd0248764f1997634f15a75fbc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35228
I20260926 11:35:23.416268 2687 raft_consensus.cc:2258] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:23.416916 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5abd0248764f1997634f15a75fbc19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54610
I20260926 11:35:23.418242 2687 ts_tablet_manager.cc:1905] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:23.418704 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:23.419245 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:23.422080 2687 ts_tablet_manager.cc:1918] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:23.422346 2687 log.cc:1196] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/6c5abd0248764f1997634f15a75fbc19
I20260926 11:35:23.422773 2687 ts_tablet_manager.cc:1939] T 6c5abd0248764f1997634f15a75fbc19 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting consensus metadata
I20260926 11:35:23.424439 2125 catalog_manager.cc:4841] TS 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229): tablet 6c5abd0248764f1997634f15a75fbc19 (table test-table [id=88509bfc06c94c5cb990c0fc8dfb56a3]) successfully deleted
I20260926 11:35:23.425742 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd65c601b1145a797da847ddcb226d8 (DEFAULT_TABLE table=test-table [id=0d7c294781e1439db64917ce2d4b8535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:23.426095 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd65c601b1145a797da847ddcb226d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:23.427855 2688 tablet_replica.cc:330] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: stopping tablet replica
I20260926 11:35:23.428288 2688 raft_consensus.cc:2229] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:23.428592 2688 raft_consensus.cc:2258] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:23.429034 2689 tablet_replica.cc:330] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: stopping tablet replica
I20260926 11:35:23.429450 2647 tablet_bootstrap.cc:492] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: Bootstrap starting.
I20260926 11:35:23.429450 2689 raft_consensus.cc:2229] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:23.429894 2689 raft_consensus.cc:2258] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:23.429884 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd65c601b1145a797da847ddcb226d8 (DEFAULT_TABLE table=test-table [id=0d7c294781e1439db64917ce2d4b8535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:23.430224 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd65c601b1145a797da847ddcb226d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:23.430775 2647 tablet_bootstrap.cc:654] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:23.431785 2689 ts_tablet_manager.cc:1905] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:23.432011 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd65c601b1145a797da847ddcb226d8 (DEFAULT_TABLE table=test-table [id=0d7c294781e1439db64917ce2d4b8535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:23.432318 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd65c601b1145a797da847ddcb226d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:23.433475 2645 tablet_bootstrap.cc:492] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: Bootstrap starting.
I20260926 11:35:23.434489 2645 tablet_bootstrap.cc:654] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:23.436447 2645 tablet_bootstrap.cc:492] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: No bootstrap required, opened a new log
I20260926 11:35:23.436558 2645 ts_tablet_manager.cc:1397] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 11:35:23.437065 2645 raft_consensus.cc:348] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.437192 2645 raft_consensus.cc:374] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:23.437228 2645 raft_consensus.cc:729] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c70782a6e9641f998a1c3f88baaad0e, State: Initialized, Role: FOLLOWER
I20260926 11:35:23.437362 2645 consensus_queue.cc:260] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.437463 2646 tablet_bootstrap.cc:492] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: Bootstrap starting.
I20260926 11:35:23.437642 2645 ts_tablet_manager.cc:1428] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:23.438609 2646 tablet_bootstrap.cc:654] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:23.440590 2647 tablet_bootstrap.cc:492] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: No bootstrap required, opened a new log
I20260926 11:35:23.440708 2647 ts_tablet_manager.cc:1397] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260926 11:35:23.440915 2646 tablet_bootstrap.cc:492] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: No bootstrap required, opened a new log
I20260926 11:35:23.441017 2646 ts_tablet_manager.cc:1397] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 11:35:23.441243 2647 raft_consensus.cc:348] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.441372 2647 raft_consensus.cc:374] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:23.441409 2647 raft_consensus.cc:729] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89132af77eb644c49f88e7198c025b5a, State: Initialized, Role: FOLLOWER
I20260926 11:35:23.441538 2689 ts_tablet_manager.cc:1918] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:23.441670 2689 log.cc:1196] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/6c5abd0248764f1997634f15a75fbc19
I20260926 11:35:23.441728 2646 raft_consensus.cc:348] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.441892 2646 raft_consensus.cc:374] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:23.441963 2646 raft_consensus.cc:729] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1839855de2c4fa0ab1232699126d88a, State: Initialized, Role: FOLLOWER
I20260926 11:35:23.441823 2647 consensus_queue.cc:260] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.442164 2689 ts_tablet_manager.cc:1939] T 6c5abd0248764f1997634f15a75fbc19 P e1839855de2c4fa0ab1232699126d88a: Deleting consensus metadata
I20260926 11:35:23.443109 2647 ts_tablet_manager.cc:1428] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 11:35:23.443663 2125 catalog_manager.cc:4841] TS e1839855de2c4fa0ab1232699126d88a (127.0.98.1:40627): tablet 6c5abd0248764f1997634f15a75fbc19 (table test-table [id=88509bfc06c94c5cb990c0fc8dfb56a3]) successfully deleted
I20260926 11:35:23.444233 2646 consensus_queue.cc:260] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.447510 2646 ts_tablet_manager.cc:1428] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 11:35:23.449834 2688 ts_tablet_manager.cc:1905] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:23.453291 2688 ts_tablet_manager.cc:1918] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:23.453428 2688 log.cc:1196] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/6c5abd0248764f1997634f15a75fbc19
I20260926 11:35:23.453742 2688 ts_tablet_manager.cc:1939] T 6c5abd0248764f1997634f15a75fbc19 P 89132af77eb644c49f88e7198c025b5a: Deleting consensus metadata
I20260926 11:35:23.455343 2123 catalog_manager.cc:4841] TS 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967): tablet 6c5abd0248764f1997634f15a75fbc19 (table test-table [id=88509bfc06c94c5cb990c0fc8dfb56a3]) successfully deleted
I20260926 11:35:23.616581 2651 raft_consensus.cc:482] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:23.616802 2651 raft_consensus.cc:504] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.617357 2651 leader_election.cc:288] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967), 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229)
I20260926 11:35:23.617913 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd65c601b1145a797da847ddcb226d8" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c70782a6e9641f998a1c3f88baaad0e" is_pre_election: true
I20260926 11:35:23.617937 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd65c601b1145a797da847ddcb226d8" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a" is_pre_election: true
I20260926 11:35:23.618098 2576 raft_consensus.cc:2454] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 0.
I20260926 11:35:23.618120 2435 raft_consensus.cc:2454] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 0.
I20260926 11:35:23.618458 2229 leader_election.cc:302] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c70782a6e9641f998a1c3f88baaad0e, e1839855de2c4fa0ab1232699126d88a; no voters:
I20260926 11:35:23.618747 2651 raft_consensus.cc:2781] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:23.618824 2651 raft_consensus.cc:482] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:23.618884 2651 raft_consensus.cc:3037] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:23.620133 2651 raft_consensus.cc:504] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.620581 2651 leader_election.cc:288] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 election: Requested vote from peers 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967), 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229)
I20260926 11:35:23.621021 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd65c601b1145a797da847ddcb226d8" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a"
I20260926 11:35:23.621021 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd65c601b1145a797da847ddcb226d8" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c70782a6e9641f998a1c3f88baaad0e"
I20260926 11:35:23.621174 2576 raft_consensus.cc:3037] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:23.621178 2435 raft_consensus.cc:3037] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:23.622435 2435 raft_consensus.cc:2454] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 1.
I20260926 11:35:23.622490 2576 raft_consensus.cc:2454] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 1.
I20260926 11:35:23.622776 2227 leader_election.cc:302] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89132af77eb644c49f88e7198c025b5a, e1839855de2c4fa0ab1232699126d88a; no voters:
I20260926 11:35:23.623001 2651 raft_consensus.cc:2781] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:23.623106 2651 raft_consensus.cc:686] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 1 LEADER]: Becoming Leader. State: Replica: e1839855de2c4fa0ab1232699126d88a, State: Running, Role: LEADER
I20260926 11:35:23.623255 2651 consensus_queue.cc:237] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:23.625175 2138 catalog_manager.cc:5495] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1839855de2c4fa0ab1232699126d88a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:23.656210 2576 raft_consensus.cc:1261] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Refusing update from remote peer e1839855de2c4fa0ab1232699126d88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:23.656277 2435 raft_consensus.cc:1261] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Refusing update from remote peer e1839855de2c4fa0ab1232699126d88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:23.656771 2678 consensus_queue.cc:1035] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:23.656980 2651 consensus_queue.cc:1035] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 11:35:23.675927 2671 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 11:35:25.256453 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:25.256661 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:25.260349 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: Sending DeleteTablet for 3 replicas of tablet 0dd65c601b1145a797da847ddcb226d8
I20260926 11:35:25.261292 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd65c601b1145a797da847ddcb226d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54610
I20260926 11:35:25.261314 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd65c601b1145a797da847ddcb226d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35228
I20260926 11:35:25.261408 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd65c601b1145a797da847ddcb226d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37566
I20260926 11:35:25.261790 2715 tablet_replica.cc:330] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: stopping tablet replica
I20260926 11:35:25.261963 2715 raft_consensus.cc:2229] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:25.262128 2715 raft_consensus.cc:2258] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:25.262241 2717 tablet_replica.cc:330] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: stopping tablet replica
I20260926 11:35:25.262454 2717 raft_consensus.cc:2229] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:25.262738 2717 raft_consensus.cc:2258] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:25.262898 2716 tablet_replica.cc:330] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: stopping tablet replica
I20260926 11:35:25.263129 2716 raft_consensus.cc:2229] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:25.263250 2715 ts_tablet_manager.cc:1905] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:25.263439 2716 raft_consensus.cc:2258] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:25.263751 2717 ts_tablet_manager.cc:1905] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:25.264508 2716 ts_tablet_manager.cc:1905] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:25.266247 2715 ts_tablet_manager.cc:1918] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:35:25.266340 2715 log.cc:1196] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/0dd65c601b1145a797da847ddcb226d8
I20260926 11:35:25.266642 2715 ts_tablet_manager.cc:1939] T 0dd65c601b1145a797da847ddcb226d8 P 89132af77eb644c49f88e7198c025b5a: Deleting consensus metadata
I20260926 11:35:25.266806 2717 ts_tablet_manager.cc:1918] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:35:25.266916 2717 log.cc:1196] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/0dd65c601b1145a797da847ddcb226d8
I20260926 11:35:25.267190 2717 ts_tablet_manager.cc:1939] T 0dd65c601b1145a797da847ddcb226d8 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting consensus metadata
I20260926 11:35:25.267854 2123 catalog_manager.cc:4841] TS 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967): tablet 0dd65c601b1145a797da847ddcb226d8 (table test-table [id=0d7c294781e1439db64917ce2d4b8535]) successfully deleted
I20260926 11:35:25.267997 2125 catalog_manager.cc:4841] TS 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229): tablet 0dd65c601b1145a797da847ddcb226d8 (table test-table [id=0d7c294781e1439db64917ce2d4b8535]) successfully deleted
I20260926 11:35:25.269649 2716 ts_tablet_manager.cc:1918] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:35:25.269758 2716 log.cc:1196] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/0dd65c601b1145a797da847ddcb226d8
I20260926 11:35:25.270102 2716 ts_tablet_manager.cc:1939] T 0dd65c601b1145a797da847ddcb226d8 P e1839855de2c4fa0ab1232699126d88a: Deleting consensus metadata
I20260926 11:35:25.271189 2125 catalog_manager.cc:4841] TS e1839855de2c4fa0ab1232699126d88a (127.0.98.1:40627): tablet 0dd65c601b1145a797da847ddcb226d8 (table test-table [id=0d7c294781e1439db64917ce2d4b8535]) successfully deleted
Sep 26 11:35:25 dist-test-slave-4pw5 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422525, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:35:25 dist-test-slave-4pw5 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422525, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:25.297715 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.286967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.0.98.62:40935)
0926 11:35:25.287425 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:25.287435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:25.287454 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:35:25.287521 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:25.287524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:25.287700 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:25.287809 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:25.288699 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.289375 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.290351 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.290595 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.290685 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.293717 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:25.293721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:25.293724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:25.293761 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:25.295818 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:25.296372 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:25.296375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:25.296376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:25.296470 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:25.296749 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:25.296751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:25.296753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:25.296970 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:25.297084 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:25.297325 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 11:35:25.297459 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":133,"threads_started":1}
I20260926 11:35:25.301412 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:25.302057 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:25.308394 2275 tablet_service.cc:1467] Processing CreateTablet for tablet d19fe8c73fc14b6884d7e1e920788315 (DEFAULT_TABLE table=test-table [id=5f38e58f6a964a8fbad2a04d1c356db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:25.308537 2416 tablet_service.cc:1467] Processing CreateTablet for tablet d19fe8c73fc14b6884d7e1e920788315 (DEFAULT_TABLE table=test-table [id=5f38e58f6a964a8fbad2a04d1c356db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:25.308583 2556 tablet_service.cc:1467] Processing CreateTablet for tablet d19fe8c73fc14b6884d7e1e920788315 (DEFAULT_TABLE table=test-table [id=5f38e58f6a964a8fbad2a04d1c356db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:25.308797 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19fe8c73fc14b6884d7e1e920788315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:25.308923 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19fe8c73fc14b6884d7e1e920788315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:25.308923 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19fe8c73fc14b6884d7e1e920788315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:25.311553 2726 tablet_bootstrap.cc:492] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: Bootstrap starting.
I20260926 11:35:25.312172 2727 tablet_bootstrap.cc:492] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: Bootstrap starting.
I20260926 11:35:25.312656 2726 tablet_bootstrap.cc:654] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:25.313203 2728 tablet_bootstrap.cc:492] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: Bootstrap starting.
I20260926 11:35:25.313472 2727 tablet_bootstrap.cc:654] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:25.314317 2726 tablet_bootstrap.cc:492] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: No bootstrap required, opened a new log
I20260926 11:35:25.314376 2728 tablet_bootstrap.cc:654] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:25.314445 2726 ts_tablet_manager.cc:1397] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 11:35:25.315054 2726 raft_consensus.cc:348] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.315205 2726 raft_consensus.cc:374] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:25.315246 2726 raft_consensus.cc:729] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1839855de2c4fa0ab1232699126d88a, State: Initialized, Role: FOLLOWER
I20260926 11:35:25.315390 2726 consensus_queue.cc:260] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.315836 2727 tablet_bootstrap.cc:492] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: No bootstrap required, opened a new log
I20260926 11:35:25.315958 2727 ts_tablet_manager.cc:1397] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 11:35:25.316339 2726 ts_tablet_manager.cc:1428] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 11:35:25.316572 2727 raft_consensus.cc:348] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.316725 2727 raft_consensus.cc:374] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:25.316757 2727 raft_consensus.cc:729] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c70782a6e9641f998a1c3f88baaad0e, State: Initialized, Role: FOLLOWER
I20260926 11:35:25.316960 2727 consensus_queue.cc:260] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.317474 2727 ts_tablet_manager.cc:1428] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 11:35:25.318172 2728 tablet_bootstrap.cc:492] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: No bootstrap required, opened a new log
I20260926 11:35:25.318292 2728 ts_tablet_manager.cc:1397] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 11:35:25.318885 2728 raft_consensus.cc:348] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.319056 2728 raft_consensus.cc:374] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:25.319092 2728 raft_consensus.cc:729] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89132af77eb644c49f88e7198c025b5a, State: Initialized, Role: FOLLOWER
I20260926 11:35:25.319257 2728 consensus_queue.cc:260] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.319770 2728 ts_tablet_manager.cc:1428] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 11:35:25.340444 2732 raft_consensus.cc:482] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:25.340606 2732 raft_consensus.cc:504] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.341830 2732 leader_election.cc:288] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1839855de2c4fa0ab1232699126d88a (127.0.98.1:40627), 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967)
I20260926 11:35:25.348361 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.342351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49279 (local address 127.0.98.2:35967)
0926 11:35:25.342731 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:25.342738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:25.343032 (+ 294us) server_negotiation.cc:407] Connection header received
0926 11:35:25.343461 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:25.343463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:25.343501 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:25.343655 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:25.343960 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.344777 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.346842 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.347562 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.347723 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.347793 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:25.348092 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 11:35:25.348189 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":169,"threads_started":1}
I20260926 11:35:25.348367 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.342268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35967 (local address 127.0.98.3:49279)
0926 11:35:25.342862 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:25.342883 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:25.343076 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:25.343687 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:25.343692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:25.343717 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:25.343846 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:25.343853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.344936 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.344939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:25.346688 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:25.346694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.347808 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.347811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:25.347919 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.347951 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 11:35:25.348026 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 11:35:25.348113 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":163,"spinlock_wait_cycles":14848,"thread_start_us":220,"threads_started":1}
I20260926 11:35:25.348562 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.342351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59029 (local address 127.0.98.1:40627)
0926 11:35:25.342850 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:25.342857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:25.342996 (+ 139us) server_negotiation.cc:407] Connection header received
0926 11:35:25.343193 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:25.343196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:25.343243 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:25.343344 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:25.343757 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.344629 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.346978 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.347618 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.347720 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.347793 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:25.348266 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 11:35:25.348401 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":170,"threads_started":1}
I20260926 11:35:25.348913 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fe8c73fc14b6884d7e1e920788315" candidate_uuid: "8c70782a6e9641f998a1c3f88baaad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a" is_pre_election: true
I20260926 11:35:25.349107 2435 raft_consensus.cc:2454] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c70782a6e9641f998a1c3f88baaad0e in term 0.
I20260926 11:35:25.349212 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.342268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40627 (local address 127.0.98.3:59029)
0926 11:35:25.342881 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:25.342895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:25.343076 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:25.343389 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:25.343392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:25.343409 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:25.343610 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:25.343622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.344878 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.344883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:25.346678 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:25.346693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.347915 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.347920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:25.348070 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.348109 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 11:35:25.348201 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 11:35:25.348311 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":239,"spinlock_wait_cycles":32256,"thread_start_us":180,"threads_started":1}
I20260926 11:35:25.349624 2507 leader_election.cc:302] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89132af77eb644c49f88e7198c025b5a, 8c70782a6e9641f998a1c3f88baaad0e; no voters:
I20260926 11:35:25.349815 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fe8c73fc14b6884d7e1e920788315" candidate_uuid: "8c70782a6e9641f998a1c3f88baaad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1839855de2c4fa0ab1232699126d88a" is_pre_election: true
I20260926 11:35:25.350021 2732 raft_consensus.cc:2781] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:25.350102 2732 raft_consensus.cc:482] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:25.350140 2732 raft_consensus.cc:3037] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:25.350143 2296 raft_consensus.cc:2454] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c70782a6e9641f998a1c3f88baaad0e in term 0.
I20260926 11:35:25.351394 2732 raft_consensus.cc:504] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.351864 2732 leader_election.cc:288] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [CANDIDATE]: Term 1 election: Requested vote from peers e1839855de2c4fa0ab1232699126d88a (127.0.98.1:40627), 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967)
I20260926 11:35:25.352286 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fe8c73fc14b6884d7e1e920788315" candidate_uuid: "8c70782a6e9641f998a1c3f88baaad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1839855de2c4fa0ab1232699126d88a"
I20260926 11:35:25.352439 2296 raft_consensus.cc:3037] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:25.352442 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19fe8c73fc14b6884d7e1e920788315" candidate_uuid: "8c70782a6e9641f998a1c3f88baaad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a"
I20260926 11:35:25.352669 2435 raft_consensus.cc:3037] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:25.353530 2296 raft_consensus.cc:2454] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c70782a6e9641f998a1c3f88baaad0e in term 1.
I20260926 11:35:25.353876 2509 leader_election.cc:302] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c70782a6e9641f998a1c3f88baaad0e, e1839855de2c4fa0ab1232699126d88a; no voters:
I20260926 11:35:25.354087 2732 raft_consensus.cc:2781] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:25.354115 2435 raft_consensus.cc:2454] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c70782a6e9641f998a1c3f88baaad0e in term 1.
I20260926 11:35:25.354331 2732 raft_consensus.cc:686] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 LEADER]: Becoming Leader. State: Replica: 8c70782a6e9641f998a1c3f88baaad0e, State: Running, Role: LEADER
I20260926 11:35:25.355129 2732 consensus_queue.cc:237] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.358285 2139 catalog_manager.cc:5495] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8c70782a6e9641f998a1c3f88baaad0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } health_report { overall_health: HEALTHY } } }
I20260926 11:35:25.382807 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.378498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.0.98.3:40229)
0926 11:35:25.378848 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:25.378855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:25.378867 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:35:25.378954 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:25.378956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:25.379017 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:25.379167 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:25.379449 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.380072 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.380996 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.381246 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.381395 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.381719 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:25.382342 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:25.382494 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:25.382531 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 11:35:25.382599 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260926 11:35:25.396396 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.392599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57677 (local address 127.0.98.1:40627)
0926 11:35:25.392797 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:25.392803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:25.392989 (+ 186us) server_negotiation.cc:407] Connection header received
0926 11:35:25.393196 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:25.393200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:25.393267 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:25.393397 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:25.393678 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.394239 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.395160 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.395399 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.395511 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.395879 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:25.395965 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:25.396047 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:25.396134 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 11:35:25.396201 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 11:35:25.396409 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.392381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40627 (local address 127.0.98.3:57677)
0926 11:35:25.392876 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:25.392893 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:25.393069 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:25.393408 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:25.393411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:25.393418 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:25.393573 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:25.393579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.394355 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.394362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:25.395039 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:25.395047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.395524 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.395528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:25.395661 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.395720 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:25.396049 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:25.396057 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 11:35:25.396126 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 11:35:25.396219 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":167,"threads_started":1}
I20260926 11:35:25.404314 2435 raft_consensus.cc:1261] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Refusing update from remote peer 8c70782a6e9641f998a1c3f88baaad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:25.404512 2296 raft_consensus.cc:1261] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Refusing update from remote peer 8c70782a6e9641f998a1c3f88baaad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:25.405110 2732 consensus_queue.cc:1035] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:25.405473 2738 consensus_queue.cc:1035] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:25.429949 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.425473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42389 (local address 127.0.98.3:40229)
0926 11:35:25.425643 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:25.425648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:25.425965 (+ 317us) server_negotiation.cc:407] Connection header received
0926 11:35:25.426098 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:25.426100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:25.426154 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:25.426222 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:25.426564 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.427230 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.428752 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.429071 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.429186 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.429454 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:25.429546 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:25.429626 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:25.429714 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 11:35:25.429776 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:35:25.429953 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.425372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40229 (local address 127.0.98.1:42389)
0926 11:35:25.425838 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:25.425856 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:25.425988 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:25.426271 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:25.426275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:25.426284 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:25.426440 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:25.426447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.427777 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.427809 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:25.428620 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:25.428638 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.429187 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.429190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:25.429286 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.429328 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:25.429628 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:25.429637 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 11:35:25.429704 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 11:35:25.429814 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":115,"threads_started":1}
I20260926 11:35:25.459489 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:25.459810 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:25.463325 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: Sending DeleteTablet for 3 replicas of tablet d19fe8c73fc14b6884d7e1e920788315
I20260926 11:35:25.464318 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet d19fe8c73fc14b6884d7e1e920788315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54610
I20260926 11:35:25.464454 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet d19fe8c73fc14b6884d7e1e920788315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35228
I20260926 11:35:25.464591 2716 tablet_replica.cc:330] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: stopping tablet replica
I20260926 11:35:25.464699 2715 tablet_replica.cc:330] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: stopping tablet replica
I20260926 11:35:25.464772 2716 raft_consensus.cc:2229] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:25.464869 2715 raft_consensus.cc:2229] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:25.464954 2716 raft_consensus.cc:2258] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:25.465026 2715 raft_consensus.cc:2258] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:25.465432 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet d19fe8c73fc14b6884d7e1e920788315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37566
I20260926 11:35:25.465651 2717 tablet_replica.cc:330] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: stopping tablet replica
I20260926 11:35:25.465799 2717 raft_consensus.cc:2229] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:25.465833 2716 ts_tablet_manager.cc:1905] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:25.466204 2717 raft_consensus.cc:2258] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:25.467051 2715 ts_tablet_manager.cc:1905] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:25.468364 2717 ts_tablet_manager.cc:1905] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:25.468767 2716 ts_tablet_manager.cc:1918] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:25.468842 2716 log.cc:1196] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/d19fe8c73fc14b6884d7e1e920788315
I20260926 11:35:25.469105 2716 ts_tablet_manager.cc:1939] T d19fe8c73fc14b6884d7e1e920788315 P e1839855de2c4fa0ab1232699126d88a: Deleting consensus metadata
I20260926 11:35:25.470283 2715 ts_tablet_manager.cc:1918] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:25.470371 2715 log.cc:1196] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/d19fe8c73fc14b6884d7e1e920788315
I20260926 11:35:25.470639 2715 ts_tablet_manager.cc:1939] T d19fe8c73fc14b6884d7e1e920788315 P 89132af77eb644c49f88e7198c025b5a: Deleting consensus metadata
I20260926 11:35:25.470988 2717 ts_tablet_manager.cc:1918] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:25.471055 2717 log.cc:1196] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/d19fe8c73fc14b6884d7e1e920788315
I20260926 11:35:25.471244 2717 ts_tablet_manager.cc:1939] T d19fe8c73fc14b6884d7e1e920788315 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting consensus metadata
I20260926 11:35:25.471639 2123 catalog_manager.cc:4841] TS 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967): tablet d19fe8c73fc14b6884d7e1e920788315 (table test-table [id=5f38e58f6a964a8fbad2a04d1c356db3]) successfully deleted
I20260926 11:35:25.471818 2125 catalog_manager.cc:4841] TS e1839855de2c4fa0ab1232699126d88a (127.0.98.1:40627): tablet d19fe8c73fc14b6884d7e1e920788315 (table test-table [id=5f38e58f6a964a8fbad2a04d1c356db3]) successfully deleted
I20260926 11:35:25.473198 2125 catalog_manager.cc:4841] TS 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229): tablet d19fe8c73fc14b6884d7e1e920788315 (table test-table [id=5f38e58f6a964a8fbad2a04d1c356db3]) successfully deleted
I20260926 11:35:25.473129 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:25.473635 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:25.479202 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 05e7a50817ec481a85ea058ed4243467 (DEFAULT_TABLE table=test-table [id=8f210543f0d945cdad703e8d9768fdb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:25.479219 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 05e7a50817ec481a85ea058ed4243467 (DEFAULT_TABLE table=test-table [id=8f210543f0d945cdad703e8d9768fdb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:25.479557 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e7a50817ec481a85ea058ed4243467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:25.479605 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e7a50817ec481a85ea058ed4243467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:25.482070 2726 tablet_bootstrap.cc:492] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a: Bootstrap starting.
I20260926 11:35:25.482453 2728 tablet_bootstrap.cc:492] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: Bootstrap starting.
I20260926 11:35:25.482991 2726 tablet_bootstrap.cc:654] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:25.483168 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 05e7a50817ec481a85ea058ed4243467 (DEFAULT_TABLE table=test-table [id=8f210543f0d945cdad703e8d9768fdb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:25.483477 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e7a50817ec481a85ea058ed4243467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:25.483505 2728 tablet_bootstrap.cc:654] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:25.484308 2726 tablet_bootstrap.cc:492] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a: No bootstrap required, opened a new log
I20260926 11:35:25.484388 2726 ts_tablet_manager.cc:1397] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 11:35:25.484807 2726 raft_consensus.cc:348] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.484961 2726 raft_consensus.cc:374] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:25.485002 2726 raft_consensus.cc:729] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1839855de2c4fa0ab1232699126d88a, State: Initialized, Role: FOLLOWER
I20260926 11:35:25.485117 2726 consensus_queue.cc:260] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.485410 2726 ts_tablet_manager.cc:1428] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 11:35:25.486280 2727 tablet_bootstrap.cc:492] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: Bootstrap starting.
I20260926 11:35:25.487319 2727 tablet_bootstrap.cc:654] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:25.488193 2728 tablet_bootstrap.cc:492] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: No bootstrap required, opened a new log
I20260926 11:35:25.488281 2728 ts_tablet_manager.cc:1397] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 11:35:25.488858 2728 raft_consensus.cc:348] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.488991 2728 raft_consensus.cc:374] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:25.489029 2728 raft_consensus.cc:729] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89132af77eb644c49f88e7198c025b5a, State: Initialized, Role: FOLLOWER
I20260926 11:35:25.489168 2728 consensus_queue.cc:260] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.489329 2727 tablet_bootstrap.cc:492] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: No bootstrap required, opened a new log
I20260926 11:35:25.489404 2727 ts_tablet_manager.cc:1397] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:25.489459 2728 ts_tablet_manager.cc:1428] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:35:25.489804 2727 raft_consensus.cc:348] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.489945 2727 raft_consensus.cc:374] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:25.489982 2727 raft_consensus.cc:729] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c70782a6e9641f998a1c3f88baaad0e, State: Initialized, Role: FOLLOWER
I20260926 11:35:25.490134 2727 consensus_queue.cc:260] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.490407 2727 ts_tablet_manager.cc:1428] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:35:25.583621 2698 raft_consensus.cc:482] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:25.583798 2698 raft_consensus.cc:504] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.584324 2698 leader_election.cc:288] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967), 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229)
I20260926 11:35:25.584810 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e7a50817ec481a85ea058ed4243467" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a" is_pre_election: true
I20260926 11:35:25.585021 2435 raft_consensus.cc:2454] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 0.
I20260926 11:35:25.585063 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e7a50817ec481a85ea058ed4243467" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c70782a6e9641f998a1c3f88baaad0e" is_pre_election: true
I20260926 11:35:25.585306 2576 raft_consensus.cc:2454] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 0.
I20260926 11:35:25.585403 2227 leader_election.cc:302] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89132af77eb644c49f88e7198c025b5a, e1839855de2c4fa0ab1232699126d88a; no voters:
I20260926 11:35:25.585678 2698 raft_consensus.cc:2781] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:25.585739 2698 raft_consensus.cc:482] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:25.585791 2698 raft_consensus.cc:3037] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:25.587095 2698 raft_consensus.cc:504] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.587540 2698 leader_election.cc:288] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 election: Requested vote from peers 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967), 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229)
I20260926 11:35:25.588069 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e7a50817ec481a85ea058ed4243467" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89132af77eb644c49f88e7198c025b5a"
I20260926 11:35:25.588218 2435 raft_consensus.cc:3037] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:25.588142 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e7a50817ec481a85ea058ed4243467" candidate_uuid: "e1839855de2c4fa0ab1232699126d88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c70782a6e9641f998a1c3f88baaad0e"
I20260926 11:35:25.588333 2576 raft_consensus.cc:3037] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:25.589567 2435 raft_consensus.cc:2454] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 1.
I20260926 11:35:25.589954 2227 leader_election.cc:302] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89132af77eb644c49f88e7198c025b5a, e1839855de2c4fa0ab1232699126d88a; no voters:
I20260926 11:35:25.590137 2576 raft_consensus.cc:2454] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1839855de2c4fa0ab1232699126d88a in term 1.
I20260926 11:35:25.590157 2698 raft_consensus.cc:2781] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:25.590310 2698 raft_consensus.cc:686] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 1 LEADER]: Becoming Leader. State: Replica: e1839855de2c4fa0ab1232699126d88a, State: Running, Role: LEADER
I20260926 11:35:25.590479 2698 consensus_queue.cc:237] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } }
I20260926 11:35:25.592944 2138 catalog_manager.cc:5495] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1839855de2c4fa0ab1232699126d88a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e1839855de2c4fa0ab1232699126d88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1839855de2c4fa0ab1232699126d88a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:25.629204 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:25.624997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.0.98.1:40627)
0926 11:35:25.625152 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:25.625156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:25.625177 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:35:25.625347 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:25.625350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:25.625413 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:25.625505 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:25.625765 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.626632 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.627719 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:25.628175 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:25.628328 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:25.628604 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:25.628733 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:25.628813 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:25.628915 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 11:35:25.628989 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:35:25.638103 2576 raft_consensus.cc:1261] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Refusing update from remote peer e1839855de2c4fa0ab1232699126d88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:25.638428 2435 raft_consensus.cc:1261] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Refusing update from remote peer e1839855de2c4fa0ab1232699126d88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:25.638909 2698 consensus_queue.cc:1035] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89132af77eb644c49f88e7198c025b5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:25.639194 2703 consensus_queue.cc:1035] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c70782a6e9641f998a1c3f88baaad0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 11:35:25.653630 2696 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 11:35:27.224684 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:27.224858 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:27.228247 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 230eee5d58e349db95a87a8e72078fac: Sending DeleteTablet for 3 replicas of tablet 05e7a50817ec481a85ea058ed4243467
I20260926 11:35:27.229146 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e7a50817ec481a85ea058ed4243467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54610
I20260926 11:35:27.229235 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e7a50817ec481a85ea058ed4243467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37566
I20260926 11:35:27.229241 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e7a50817ec481a85ea058ed4243467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35228
I20260926 11:35:27.229794 2779 tablet_replica.cc:330] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: stopping tablet replica
I20260926 11:35:27.229815 2778 tablet_replica.cc:330] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a: stopping tablet replica
I20260926 11:35:27.229985 2779 raft_consensus.cc:2229] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:27.230129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 2211
I20260926 11:35:27.229997 2778 raft_consensus.cc:2229] T 05e7a50817ec481a85ea058ed4243467 P e1839855de2c4fa0ab1232699126d88a [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:27.230208 2779 raft_consensus.cc:2258] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:27.232564 2777 tablet_replica.cc:330] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: stopping tablet replica
I20260926 11:35:27.232746 2777 raft_consensus.cc:2229] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:27.232914 2777 raft_consensus.cc:2258] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:27.233361 2779 ts_tablet_manager.cc:1905] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:27.233750 2777 ts_tablet_manager.cc:1905] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:27.235878 2779 ts_tablet_manager.cc:1918] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:35:27.235986 2779 log.cc:1196] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/05e7a50817ec481a85ea058ed4243467
I20260926 11:35:27.236313 2779 ts_tablet_manager.cc:1939] T 05e7a50817ec481a85ea058ed4243467 P 89132af77eb644c49f88e7198c025b5a: Deleting consensus metadata
I20260926 11:35:27.237124 2777 ts_tablet_manager.cc:1918] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:35:27.237226 2777 log.cc:1196] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/05e7a50817ec481a85ea058ed4243467
I20260926 11:35:27.237174 2123 catalog_manager.cc:4841] TS 89132af77eb644c49f88e7198c025b5a (127.0.98.2:35967): tablet 05e7a50817ec481a85ea058ed4243467 (table test-table [id=8f210543f0d945cdad703e8d9768fdb5]) successfully deleted
I20260926 11:35:27.237506 2777 ts_tablet_manager.cc:1939] T 05e7a50817ec481a85ea058ed4243467 P 8c70782a6e9641f998a1c3f88baaad0e: Deleting consensus metadata
I20260926 11:35:27.238648 2125 catalog_manager.cc:4841] TS 8c70782a6e9641f998a1c3f88baaad0e (127.0.98.3:40229): tablet 05e7a50817ec481a85ea058ed4243467 (table test-table [id=8f210543f0d945cdad703e8d9768fdb5]) successfully deleted
W20260926 11:35:27.240439 2125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40627 (error 108)
I20260926 11:35:27.240561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 2351
I20260926 11:35:27.244820 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:27.244313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40627 (local address 127.0.0.1:35020)
0926 11:35:27.244608 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:27.244707 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":138,"threads_started":1}
W20260926 11:35:27.245183 2125 catalog_manager.cc:4584] TS e1839855de2c4fa0ab1232699126d88a (127.0.98.1:40627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05e7a50817ec481a85ea058ed4243467: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40627: connect: Connection refused (error 111)
I20260926 11:35:27.248164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 2491
I20260926 11:35:27.257730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 2107
2026-09-26T11:35:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6618 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:27 dist-test-slave-4pw5 krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:27 dist-test-slave-4pw5 krb5kdc[2784](info): set up 2 sockets
Sep 26 11:35:27 dist-test-slave-4pw5 krb5kdc[2784](info): commencing operation
krb5kdc: starting...
W20260926 11:35:28.355958 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:28 dist-test-slave-4pw5 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:28.492642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45973
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:28.610251 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:28.610563 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:28.610606 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:28.614972 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:28.615075 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:28.615096 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:28.615114 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:28.615131 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:28.621137 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:28.622291 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:28.623320 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:28.634027 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:28.634060 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:28.634051 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:28.634040 2797 server_base.cc:1034] running on GCE node
I20260926 11:35:28.635138 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:28.636569 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:28.637814 2797 hybrid_clock.cc:648] HybridClock initialized: now 1790422528637774 us; error 59 us; skew 500 ppm
Sep 26 11:35:28 dist-test-slave-4pw5 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:28.641629 2797 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:28.642890 2797 webserver.cc:460] Webserver started at http://127.0.98.62:38827/ using document root <none> and password file <none>
I20260926 11:35:28.643551 2797 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:28.643723 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:28.643976 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:28.646281 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "77fb25d1411246db89fbf1b5b74d6b50"
format_stamp: "Formatted at 2026-09-26 11:35:28 on dist-test-slave-4pw5"
server_key: "dcade111fb6905ff64d6a2779a633abb"
server_key_iv: "f13fc7753cad35a19fdf55c611dc2ac1"
server_key_version: "encryptionkey@0"
I20260926 11:35:28.646912 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "77fb25d1411246db89fbf1b5b74d6b50"
format_stamp: "Formatted at 2026-09-26 11:35:28 on dist-test-slave-4pw5"
server_key: "dcade111fb6905ff64d6a2779a633abb"
server_key_iv: "f13fc7753cad35a19fdf55c611dc2ac1"
server_key_version: "encryptionkey@0"
I20260926 11:35:28.650998 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:35:28.653712 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:28.655023 2797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:35:28.655192 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "77fb25d1411246db89fbf1b5b74d6b50"
format_stamp: "Formatted at 2026-09-26 11:35:28 on dist-test-slave-4pw5"
server_key: "dcade111fb6905ff64d6a2779a633abb"
server_key_iv: "f13fc7753cad35a19fdf55c611dc2ac1"
server_key_version: "encryptionkey@0"
I20260926 11:35:28.655326 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:28.665751 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:28.669710 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:28.669965 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:28.678627 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45973
I20260926 11:35:28.678634 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45973 every 8 connection(s)
I20260926 11:35:28.679847 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:28.683085 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:28.689357 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50: Bootstrap starting.
I20260926 11:35:28.689833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 2797
I20260926 11:35:28.689949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:28.690230 392 external_mini_cluster.cc:1428] Setting key f687cb3bd1432fd54efc885db0491091
I20260926 11:35:28.692541 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:28.693361 2865 log.cc:824] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:28.695386 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50: No bootstrap required, opened a new log
Sep 26 11:35:28 dist-test-slave-4pw5 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:28.700062 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb25d1411246db89fbf1b5b74d6b50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } }
I20260926 11:35:28.700420 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:28.700512 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fb25d1411246db89fbf1b5b74d6b50, State: Initialized, Role: FOLLOWER
I20260926 11:35:28.701081 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb25d1411246db89fbf1b5b74d6b50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } }
I20260926 11:35:28.701217 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:28.701313 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:28.701442 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:28.702525 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb25d1411246db89fbf1b5b74d6b50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } }
I20260926 11:35:28.703109 2865 leader_election.cc:302] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77fb25d1411246db89fbf1b5b74d6b50; no voters:
I20260926 11:35:28.703555 2865 leader_election.cc:288] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:28.703725 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:28.704002 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [term 1 LEADER]: Becoming Leader. State: Replica: 77fb25d1411246db89fbf1b5b74d6b50, State: Running, Role: LEADER
I20260926 11:35:28.704421 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb25d1411246db89fbf1b5b74d6b50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } }
I20260926 11:35:28.705147 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:28.706282 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77fb25d1411246db89fbf1b5b74d6b50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb25d1411246db89fbf1b5b74d6b50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } }
I20260926 11:35:28.706414 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:28.706588 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:28.691758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.0.98.62:45973)
0926 11:35:28.692267 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:28.692282 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:28.692320 (+ 38us) server_negotiation.cc:407] Connection header received
0926 11:35:28.693038 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:28.693056 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:28.693415 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:28.693799 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:28.694653 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:28.695775 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:28.696653 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:28.697132 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:28.697308 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:28.700388 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:28.700413 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:28.700435 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:28.700481 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:28.702490 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:28.703143 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:28.703150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:28.703158 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:28.703348 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:28.703736 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:28.703739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:28.703741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:28.704330 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:28.704541 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:28.705003 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 11:35:28.705466 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":152,"threads_started":1}
I20260926 11:35:28.706736 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77fb25d1411246db89fbf1b5b74d6b50. Latest consensus state: current_term: 1 leader_uuid: "77fb25d1411246db89fbf1b5b74d6b50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fb25d1411246db89fbf1b5b74d6b50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } }
I20260926 11:35:28.707049 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50 [sys.catalog]: This master's current role is: LEADER
W20260926 11:35:28.711037 2886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:35:28.711136 2886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:35:28.711232 2879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:28.712044 2879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:28.717603 2879 catalog_manager.cc:1347] Generated new cluster ID: 959791e8e69b4b349afb39c3bf84776f
I20260926 11:35:28.717693 2879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:28.752960 2879 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:28.754005 2879 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:28.766800 2879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77fb25d1411246db89fbf1b5b74d6b50: Generated new TSK 0
I20260926 11:35:28.767633 2879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:28.808007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:28.945489 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:28.945814 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:28.945861 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:28.950301 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:28.950383 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:28.950486 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:28.956907 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:28.958159 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:28.959340 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:28.970310 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:28.970309 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:28.970407 2891 server_base.cc:1034] running on GCE node
W20260926 11:35:28.970345 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:28.971128 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:28.971859 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:28.973042 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790422528973022 us; error 35 us; skew 500 ppm
Sep 26 11:35:28 dist-test-slave-4pw5 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:28.976214 2891 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:28.977234 2891 webserver.cc:460] Webserver started at http://127.0.98.1:34307/ using document root <none> and password file <none>
I20260926 11:35:28.977855 2891 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:28.977923 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:28.978163 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:28.980526 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "f261d2c93a084b69918c538b26c5a586"
format_stamp: "Formatted at 2026-09-26 11:35:28 on dist-test-slave-4pw5"
server_key: "ca6824ebeb1ab46bcae0b5767452b282"
server_key_iv: "aa1c69f78c74af806361e1ea87c4e6cf"
server_key_version: "encryptionkey@0"
I20260926 11:35:28.981177 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f261d2c93a084b69918c538b26c5a586"
format_stamp: "Formatted at 2026-09-26 11:35:28 on dist-test-slave-4pw5"
server_key: "ca6824ebeb1ab46bcae0b5767452b282"
server_key_iv: "aa1c69f78c74af806361e1ea87c4e6cf"
server_key_version: "encryptionkey@0"
I20260926 11:35:28.985390 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:35:28.988252 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:28.989763 2891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:28.990017 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "f261d2c93a084b69918c538b26c5a586"
format_stamp: "Formatted at 2026-09-26 11:35:28 on dist-test-slave-4pw5"
server_key: "ca6824ebeb1ab46bcae0b5767452b282"
server_key_iv: "aa1c69f78c74af806361e1ea87c4e6cf"
server_key_version: "encryptionkey@0"
I20260926 11:35:28.990176 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:29.009042 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:29.013281 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:29.013543 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:29.014240 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:29.015144 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:29.015192 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:29.015270 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:29.015300 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:29.026083 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40225
I20260926 11:35:29.026135 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40225 every 8 connection(s)
I20260926 11:35:29.027081 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:29.036175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 2891
I20260926 11:35:29.036338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:29.036720 392 external_mini_cluster.cc:1428] Setting key e0420ec1c1309e41e0ca9f5c5e7898a8
Sep 26 11:35:29 dist-test-slave-4pw5 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:29.045244 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:29.029198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42105 (local address 127.0.98.62:45973)
0926 11:35:29.029372 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:29.029375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:29.030230 (+ 855us) server_negotiation.cc:407] Connection header received
0926 11:35:29.031875 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:29.031879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:29.031945 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:29.032089 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:29.033414 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.034103 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.036249 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.036604 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.036727 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:29.039922 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:29.039930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:29.039936 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:29.040020 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:29.042042 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:29.043049 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:29.043053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:29.043055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:29.043161 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:29.043743 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:29.043746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:29.043747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:29.044009 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:29.044104 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:29.044937 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 11:35:29.045107 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 11:35:29.046833 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:29.029577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.1:42105)
0926 11:35:29.030065 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:29.030120 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:29.031515 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:29.032295 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:29.032303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:29.032708 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:29.033245 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:29.033260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.034288 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.034297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:29.036044 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:29.036061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.036783 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.036788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:29.036988 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:29.037848 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:29.037871 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:29.039727 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:29.042228 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:29.042240 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:29.042265 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:29.042866 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:29.043282 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:29.043288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:29.043292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:29.043545 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:29.044142 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:29.044153 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:29.044597 (+ 444us) client_negotiation.cc:769] Sending connection context
0926 11:35:29.044845 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 11:35:29.045268 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:35:29.048358 3020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45973
I20260926 11:35:29.048619 3020 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:29.049201 3020 heartbeater.cc:511] Master 127.0.98.62:45973 requested a full tablet report, sending...
I20260926 11:35:29.050874 2829 ts_manager.cc:194] Registered new tserver with Master: f261d2c93a084b69918c538b26c5a586 (127.0.98.1:40225)
I20260926 11:35:29.052253 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260926 11:35:29.167865 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.129s user 0.000s sys 0.003s
I20260926 11:35:29.168474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:29.293843 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:29.294268 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:29.294320 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:29.299690 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:29.299770 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:29.299885 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:29.305912 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:29.307209 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:29.308367 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:29.319063 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:29.319073 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:29.319169 3025 server_base.cc:1034] running on GCE node
W20260926 11:35:29.319087 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:29.319844 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:29.320565 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:29.321786 3025 hybrid_clock.cc:648] HybridClock initialized: now 1790422529321758 us; error 45 us; skew 500 ppm
Sep 26 11:35:29 dist-test-slave-4pw5 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:29.325042 3025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:35:29.326002 3025 webserver.cc:460] Webserver started at http://127.0.98.2:37055/ using document root <none> and password file <none>
I20260926 11:35:29.326566 3025 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:29.326630 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:29.326804 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:29.329025 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "08e923a7c2e34ba49728bc8ac07a5de6"
format_stamp: "Formatted at 2026-09-26 11:35:29 on dist-test-slave-4pw5"
server_key: "1d278494050bc0198a59d37ea24eb774"
server_key_iv: "c79f26d9f318b21819d25395ae1eb9bf"
server_key_version: "encryptionkey@0"
I20260926 11:35:29.329596 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "08e923a7c2e34ba49728bc8ac07a5de6"
format_stamp: "Formatted at 2026-09-26 11:35:29 on dist-test-slave-4pw5"
server_key: "1d278494050bc0198a59d37ea24eb774"
server_key_iv: "c79f26d9f318b21819d25395ae1eb9bf"
server_key_version: "encryptionkey@0"
I20260926 11:35:29.333956 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:35:29.336687 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:29.338057 3025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:29.338164 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "08e923a7c2e34ba49728bc8ac07a5de6"
format_stamp: "Formatted at 2026-09-26 11:35:29 on dist-test-slave-4pw5"
server_key: "1d278494050bc0198a59d37ea24eb774"
server_key_iv: "c79f26d9f318b21819d25395ae1eb9bf"
server_key_version: "encryptionkey@0"
I20260926 11:35:29.338264 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:29.358506 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:29.361829 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:29.361994 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:29.362633 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:29.363561 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:29.363757 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:29.363806 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:29.363827 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:29.373394 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43953
I20260926 11:35:29.373413 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43953 every 8 connection(s)
I20260926 11:35:29.374377 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:29.375733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 3025
I20260926 11:35:29.375828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:29.376119 392 external_mini_cluster.cc:1428] Setting key 370daebe2f21ea33a073f95488649d5e
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 26 11:35:29 dist-test-slave-4pw5 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:29.395668 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:29.376553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46589 (local address 127.0.98.62:45973)
0926 11:35:29.376727 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:29.376732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:29.377817 (+ 1085us) server_negotiation.cc:407] Connection header received
0926 11:35:29.379900 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:29.379903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:29.379953 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:29.380017 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:29.381758 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.382800 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.384849 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.385099 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.385199 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:29.391271 (+ 6072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:29.391280 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:29.391284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:29.391330 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:29.393174 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:29.393844 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:29.393848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:29.393850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:29.393944 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:29.394347 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:29.394351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:29.394353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:29.394583 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:29.394702 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:29.395295 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 11:35:29.395439 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 11:35:29.396651 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:29.377105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.2:46589)
0926 11:35:29.377617 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:29.377724 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:29.379581 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:29.380258 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:29.380273 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:29.380816 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:29.381552 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:29.381578 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.383008 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.383014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:29.384690 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:29.384704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.385283 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.385291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:29.385526 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:29.386213 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:29.386237 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:29.390981 (+ 4744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:29.393349 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:29.393357 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:29.393385 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:29.393712 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:29.394049 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:29.394053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:29.394056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:29.394227 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:29.394704 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:29.394715 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:29.395014 (+ 299us) client_negotiation.cc:769] Sending connection context
0926 11:35:29.395240 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 11:35:29.395652 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":144,"threads_started":1}
I20260926 11:35:29.397975 3154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45973
I20260926 11:35:29.398262 3154 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:29.398854 3154 heartbeater.cc:511] Master 127.0.98.62:45973 requested a full tablet report, sending...
I20260926 11:35:29.400137 2829 ts_manager.cc:194] Registered new tserver with Master: 08e923a7c2e34ba49728bc8ac07a5de6 (127.0.98.2:43953)
I20260926 11:35:29.400769 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:35:29.408825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:29.527782 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:29.528076 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:29.528177 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:29.533051 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:29.533152 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:29.533226 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:29.539529 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:29.540850 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:29.542016 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:29.552990 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:29.553004 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:29.553045 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:29.553387 3159 server_base.cc:1034] running on GCE node
I20260926 11:35:29.553975 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:29.554765 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:29.555969 3159 hybrid_clock.cc:648] HybridClock initialized: now 1790422529555933 us; error 66 us; skew 500 ppm
Sep 26 11:35:29 dist-test-slave-4pw5 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:29.565157 3159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:35:29.566332 3159 webserver.cc:460] Webserver started at http://127.0.98.3:34885/ using document root <none> and password file <none>
I20260926 11:35:29.567031 3159 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:29.567104 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:29.567359 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:29.569739 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "918c3b19837042bda8c271330f77bf1c"
format_stamp: "Formatted at 2026-09-26 11:35:29 on dist-test-slave-4pw5"
server_key: "b2379022ed76eec1d3f9d491f8826974"
server_key_iv: "036d4d2d76a869eb52886a6ac1ae4171"
server_key_version: "encryptionkey@0"
I20260926 11:35:29.570405 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "918c3b19837042bda8c271330f77bf1c"
format_stamp: "Formatted at 2026-09-26 11:35:29 on dist-test-slave-4pw5"
server_key: "b2379022ed76eec1d3f9d491f8826974"
server_key_iv: "036d4d2d76a869eb52886a6ac1ae4171"
server_key_version: "encryptionkey@0"
I20260926 11:35:29.574980 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 11:35:29.578002 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:29.579550 3159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:29.579783 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "918c3b19837042bda8c271330f77bf1c"
format_stamp: "Formatted at 2026-09-26 11:35:29 on dist-test-slave-4pw5"
server_key: "b2379022ed76eec1d3f9d491f8826974"
server_key_iv: "036d4d2d76a869eb52886a6ac1ae4171"
server_key_version: "encryptionkey@0"
I20260926 11:35:29.579906 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:29.596568 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:29.599995 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:29.600176 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:29.600736 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:29.601580 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:29.601631 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:29.601675 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:29.601691 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:29.611408 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44865
I20260926 11:35:29.611441 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44865 every 8 connection(s)
I20260926 11:35:29.612394 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:29.616194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 3159
I20260926 11:35:29.616294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:29.616580 392 external_mini_cluster.cc:1428] Setting key 981dba08c75cc4ebf9d3febbd2a8435e
Sep 26 11:35:29 dist-test-slave-4pw5 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:29.627555 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:29.614380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40945 (local address 127.0.98.62:45973)
0926 11:35:29.614522 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:29.614526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:29.615455 (+ 929us) server_negotiation.cc:407] Connection header received
0926 11:35:29.617134 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:29.617138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:29.617223 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:29.617320 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:29.619057 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.619675 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.620890 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.621139 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.621248 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:29.624033 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:29.624036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:29.624039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:29.624064 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:29.625457 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:29.625975 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:29.625981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:29.625982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:29.626064 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:29.626425 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:29.626428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:29.626430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:29.626643 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:29.626746 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:29.627264 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 11:35:29.627389 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:35:29.628438 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:29.614784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.3:40945)
0926 11:35:29.615277 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:29.615326 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:29.616847 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:29.617592 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:29.617603 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:29.618188 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:29.618918 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:29.618940 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.619807 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.619810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:29.620757 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:29.620766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:29.621306 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:29.621310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:29.621494 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:29.622134 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:29.622152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:29.623831 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:29.625574 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:29.625580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:29.625597 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:29.625869 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:29.626164 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:29.626167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:29.626170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:29.626323 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:29.626745 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:29.626752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:29.627016 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 11:35:29.627218 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 11:35:29.627564 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":110,"threads_started":1}
I20260926 11:35:29.629710 3288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45973
I20260926 11:35:29.630015 3288 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:29.630590 3288 heartbeater.cc:511] Master 127.0.98.62:45973 requested a full tablet report, sending...
I20260926 11:35:29.631836 2829 ts_manager.cc:194] Registered new tserver with Master: 918c3b19837042bda8c271330f77bf1c (127.0.98.3:44865)
I20260926 11:35:29.632388 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40945
I20260926 11:35:29.643251 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 11:35:29.657876 2868 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:29.656144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.0.98.62:45973)
0926 11:35:29.656300 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:29.656304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:29.656439 (+ 135us) server_negotiation.cc:407] Connection header received
0926 11:35:29.656609 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:29.656612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:29.656657 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:29.656768 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:29.657694 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38490: BlockingRecv error: recv got EOF from 127.0.0.1:38490 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 11:35:29.659020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 2891
I20260926 11:35:29.664717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 3025
I20260926 11:35:29.669878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 3159
I20260926 11:35:29.675288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 2797
2026-09-26T11:35:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2412 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:29 dist-test-slave-4pw5 krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:29 dist-test-slave-4pw5 krb5kdc[3300](info): set up 2 sockets
Sep 26 11:35:29 dist-test-slave-4pw5 krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260926 11:35:30.743731 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:30 dist-test-slave-4pw5 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:30.879703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42961
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:30.996248 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:30.996601 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:30.996686 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:31.000942 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:31.001048 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:31.001087 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:31.001142 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:31.001168 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:31.006866 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:31.008162 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:31.009229 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:35:31.018983 3313 server_base.cc:1034] running on GCE node
W20260926 11:35:31.019050 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:31.019187 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:31.019050 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:31.020085 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:31.021420 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:31.022676 3313 hybrid_clock.cc:648] HybridClock initialized: now 1790422531022633 us; error 55 us; skew 500 ppm
Sep 26 11:35:31 dist-test-slave-4pw5 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:31.026023 3313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:31.026983 3313 webserver.cc:460] Webserver started at http://127.0.98.62:42095/ using document root <none> and password file <none>
I20260926 11:35:31.027556 3313 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:31.027694 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:31.027889 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:31.030071 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "db2c3568b97d4fe4a7f3e7654a7653fa"
format_stamp: "Formatted at 2026-09-26 11:35:31 on dist-test-slave-4pw5"
server_key: "c49de97d66060722a6d8aec7e56c09a0"
server_key_iv: "061cb7e7ff2e2bd0f40348454f77f0b5"
server_key_version: "encryptionkey@0"
I20260926 11:35:31.030646 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "db2c3568b97d4fe4a7f3e7654a7653fa"
format_stamp: "Formatted at 2026-09-26 11:35:31 on dist-test-slave-4pw5"
server_key: "c49de97d66060722a6d8aec7e56c09a0"
server_key_iv: "061cb7e7ff2e2bd0f40348454f77f0b5"
server_key_version: "encryptionkey@0"
I20260926 11:35:31.034969 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 11:35:31.038335 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:31.040081 3313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:31.040244 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "db2c3568b97d4fe4a7f3e7654a7653fa"
format_stamp: "Formatted at 2026-09-26 11:35:31 on dist-test-slave-4pw5"
server_key: "c49de97d66060722a6d8aec7e56c09a0"
server_key_iv: "061cb7e7ff2e2bd0f40348454f77f0b5"
server_key_version: "encryptionkey@0"
I20260926 11:35:31.040372 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:31.061048 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:31.064694 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:31.064924 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:31.072789 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42961
I20260926 11:35:31.073064 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42961 every 8 connection(s)
I20260926 11:35:31.074066 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:31.076783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 3313
I20260926 11:35:31.076894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:31.077194 392 external_mini_cluster.cc:1428] Setting key eeb7c3574c2c2d088cf284edcf46238a
I20260926 11:35:31.077284 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:31.085007 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa: Bootstrap starting.
Sep 26 11:35:31 dist-test-slave-4pw5 krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:31.087486 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:31.088415 3381 log.cc:824] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:31.090981 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa: No bootstrap required, opened a new log
I20260926 11:35:31.093689 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:31.078671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.0.98.62:42961)
0926 11:35:31.079156 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:31.079176 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:31.079222 (+ 46us) server_negotiation.cc:407] Connection header received
0926 11:35:31.080232 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:31.080261 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:31.080714 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:31.081131 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:31.081996 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:31.083521 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:31.084549 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:31.085280 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:31.085446 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:31.088185 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:31.088195 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:31.088214 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:31.088256 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:31.090324 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:31.090912 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:31.090918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:31.090924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:31.091069 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:31.091458 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:31.091462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:31.091463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:31.091976 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:31.092142 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:31.092447 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 11:35:31.092774 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":137,"threads_started":1}
I20260926 11:35:31.095110 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2c3568b97d4fe4a7f3e7654a7653fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260926 11:35:31.095347 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:31.095408 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2c3568b97d4fe4a7f3e7654a7653fa, State: Initialized, Role: FOLLOWER
I20260926 11:35:31.095989 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2c3568b97d4fe4a7f3e7654a7653fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260926 11:35:31.096124 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:31.096206 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:31.096304 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:31.097255 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:35:31.097416 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2c3568b97d4fe4a7f3e7654a7653fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260926 11:35:31.097779 3381 leader_election.cc:302] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db2c3568b97d4fe4a7f3e7654a7653fa; no voters:
I20260926 11:35:31.098125 3381 leader_election.cc:288] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:31.098215 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:31.099396 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [term 1 LEADER]: Becoming Leader. State: Replica: db2c3568b97d4fe4a7f3e7654a7653fa, State: Running, Role: LEADER
I20260926 11:35:31.099503 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:31.099790 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2c3568b97d4fe4a7f3e7654a7653fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260926 11:35:31.101591 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db2c3568b97d4fe4a7f3e7654a7653fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2c3568b97d4fe4a7f3e7654a7653fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } }
I20260926 11:35:31.101715 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:31.102056 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader db2c3568b97d4fe4a7f3e7654a7653fa. Latest consensus state: current_term: 1 leader_uuid: "db2c3568b97d4fe4a7f3e7654a7653fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2c3568b97d4fe4a7f3e7654a7653fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } }
I20260926 11:35:31.102137 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa [sys.catalog]: This master's current role is: LEADER
W20260926 11:35:31.104898 3401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:35:31.105012 3401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:35:31.105118 3402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:31.106240 3402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:31.112964 3402 catalog_manager.cc:1347] Generated new cluster ID: 8bc6c3f9e9524d6e9fb3adf26ce8c8ad
I20260926 11:35:31.113186 3402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:31.120337 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:31.114074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.0.98.62:42961)
0926 11:35:31.114195 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:31.114200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:31.114501 (+ 301us) server_negotiation.cc:407] Connection header received
0926 11:35:31.114630 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:31.114634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:31.114687 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:31.114778 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:31.115430 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:31.116338 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:31.117571 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:31.117976 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:31.118082 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:31.119113 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:31.119117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:31.119120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:31.119830 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:31.119868 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:31.119889 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:31.119982 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:31.120110 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 11:35:31.120176 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:35:31.125079 3402 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:31.126260 3402 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:31.131673 3402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db2c3568b97d4fe4a7f3e7654a7653fa: Generated new TSK 0
I20260926 11:35:31.132640 3402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:35:31.145782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 3313
2026-09-26T11:35:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1469 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:31 dist-test-slave-4pw5 krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:31 dist-test-slave-4pw5 krb5kdc[3414](info): set up 2 sockets
Sep 26 11:35:31 dist-test-slave-4pw5 krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260926 11:35:32.219784 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:32 dist-test-slave-4pw5 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:32.352807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:32.468721 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:32.469053 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:32.469100 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:32.473390 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:32.473466 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:32.473487 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:32.473505 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:32.473520 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:32.479154 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:32.480285 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:32.481303 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:32.490089 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:32.490098 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:32.490098 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:32.490497 3427 server_base.cc:1034] running on GCE node
I20260926 11:35:32.491112 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:32.492411 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:32.493671 3427 hybrid_clock.cc:648] HybridClock initialized: now 1790422532493638 us; error 52 us; skew 500 ppm
Sep 26 11:35:32 dist-test-slave-4pw5 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:32.501327 3427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:32.502461 3427 webserver.cc:460] Webserver started at http://127.0.98.62:36753/ using document root <none> and password file <none>
I20260926 11:35:32.503118 3427 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:32.503221 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:32.503487 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:32.505883 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "a6639f9b19704d029c69e5b12d1092fd"
format_stamp: "Formatted at 2026-09-26 11:35:32 on dist-test-slave-4pw5"
server_key: "f4b80510a4cce938edc9ab30e7fe8925"
server_key_iv: "b4a16aef9028b0f1ff82000aabd43b9a"
server_key_version: "encryptionkey@0"
I20260926 11:35:32.506536 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6639f9b19704d029c69e5b12d1092fd"
format_stamp: "Formatted at 2026-09-26 11:35:32 on dist-test-slave-4pw5"
server_key: "f4b80510a4cce938edc9ab30e7fe8925"
server_key_iv: "b4a16aef9028b0f1ff82000aabd43b9a"
server_key_version: "encryptionkey@0"
I20260926 11:35:32.510866 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 11:35:32.513876 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:32.515283 3427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:32.515480 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "a6639f9b19704d029c69e5b12d1092fd"
format_stamp: "Formatted at 2026-09-26 11:35:32 on dist-test-slave-4pw5"
server_key: "f4b80510a4cce938edc9ab30e7fe8925"
server_key_iv: "b4a16aef9028b0f1ff82000aabd43b9a"
server_key_version: "encryptionkey@0"
I20260926 11:35:32.515653 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:32.535027 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:32.538699 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:32.538959 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:32.546923 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37995
I20260926 11:35:32.547304 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37995 every 8 connection(s)
I20260926 11:35:32.548059 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:32.550665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 3427
I20260926 11:35:32.550810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:32.551110 392 external_mini_cluster.cc:1428] Setting key de922f3a8ee6c312c7e3811acdd4a30f
I20260926 11:35:32.551520 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:32.559288 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd: Bootstrap starting.
Sep 26 11:35:32 dist-test-slave-4pw5 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:32.562071 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:32.562860 3495 log.cc:824] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:32.564981 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd: No bootstrap required, opened a new log
I20260926 11:35:32.566874 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:32.552729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.0.98.62:37995)
0926 11:35:32.553166 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:32.553192 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:32.553230 (+ 38us) server_negotiation.cc:407] Connection header received
0926 11:35:32.553900 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:32.553917 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:32.554279 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:32.554584 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:32.555637 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:32.556760 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:32.558114 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:32.558637 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:32.558847 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:32.561822 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:32.561827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:32.561843 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:32.561880 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:32.563847 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:32.564379 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:32.564385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:32.564390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:32.564511 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:32.564809 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:32.564813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:32.564815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:32.565305 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:32.565508 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:32.565821 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 11:35:32.566121 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":154,"threads_started":1}
I20260926 11:35:32.568897 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6639f9b19704d029c69e5b12d1092fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } }
I20260926 11:35:32.569198 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:32.569270 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6639f9b19704d029c69e5b12d1092fd, State: Initialized, Role: FOLLOWER
I20260926 11:35:32.569664 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6639f9b19704d029c69e5b12d1092fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } }
I20260926 11:35:32.569783 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:32.569830 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:32.569906 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:32.570833 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6639f9b19704d029c69e5b12d1092fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } }
I20260926 11:35:32.571146 3495 leader_election.cc:302] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6639f9b19704d029c69e5b12d1092fd; no voters:
I20260926 11:35:32.571472 3495 leader_election.cc:288] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:32.571664 3500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:32.571998 3500 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [term 1 LEADER]: Becoming Leader. State: Replica: a6639f9b19704d029c69e5b12d1092fd, State: Running, Role: LEADER
I20260926 11:35:32.572352 3500 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6639f9b19704d029c69e5b12d1092fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } }
I20260926 11:35:32.572875 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:32.574108 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6639f9b19704d029c69e5b12d1092fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6639f9b19704d029c69e5b12d1092fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } }
I20260926 11:35:32.574216 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:32.574584 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6639f9b19704d029c69e5b12d1092fd. Latest consensus state: current_term: 1 leader_uuid: "a6639f9b19704d029c69e5b12d1092fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6639f9b19704d029c69e5b12d1092fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } }
I20260926 11:35:32.574661 3509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:32.574677 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:32.577554 3509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:32.583918 3509 catalog_manager.cc:1347] Generated new cluster ID: 48f64b0f59c14adb83563f7f4e9aadce
I20260926 11:35:32.584008 3509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:32.605705 3509 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:32.607045 3509 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:32.620798 3509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6639f9b19704d029c69e5b12d1092fd: Generated new TSK 0
I20260926 11:35:32.622265 3509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:32.663825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:32.785218 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:32.785535 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:32.785581 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:32.789849 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:32.789919 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:32.789981 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:32.795516 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:32.796691 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:32.797698 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:32.807803 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:32.807898 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:32.807898 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:32.808014 3521 server_base.cc:1034] running on GCE node
I20260926 11:35:32.808578 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:32.809291 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:32.810496 3521 hybrid_clock.cc:648] HybridClock initialized: now 1790422532810471 us; error 38 us; skew 500 ppm
Sep 26 11:35:32 dist-test-slave-4pw5 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:32.813822 3521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:32.814739 3521 webserver.cc:460] Webserver started at http://127.0.98.1:40145/ using document root <none> and password file <none>
I20260926 11:35:32.815320 3521 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:32.815384 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:32.815618 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:32.817737 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "7ef180748fff408d8ee5141e13d775b5"
format_stamp: "Formatted at 2026-09-26 11:35:32 on dist-test-slave-4pw5"
server_key: "5f2a38452b2008cee788db22a7ba37bd"
server_key_iv: "39ec7b75e60aef04ac3db73ba9aaab8c"
server_key_version: "encryptionkey@0"
I20260926 11:35:32.818261 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7ef180748fff408d8ee5141e13d775b5"
format_stamp: "Formatted at 2026-09-26 11:35:32 on dist-test-slave-4pw5"
server_key: "5f2a38452b2008cee788db22a7ba37bd"
server_key_iv: "39ec7b75e60aef04ac3db73ba9aaab8c"
server_key_version: "encryptionkey@0"
I20260926 11:35:32.822274 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:35:32.824898 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:32.826357 3521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:35:32.826521 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "7ef180748fff408d8ee5141e13d775b5"
format_stamp: "Formatted at 2026-09-26 11:35:32 on dist-test-slave-4pw5"
server_key: "5f2a38452b2008cee788db22a7ba37bd"
server_key_iv: "39ec7b75e60aef04ac3db73ba9aaab8c"
server_key_version: "encryptionkey@0"
I20260926 11:35:32.826792 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:32.853156 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:32.856915 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:32.857168 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:32.857844 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:32.858793 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:32.858870 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:32.858942 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:32.858994 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:32.869771 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41011
I20260926 11:35:32.869799 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41011 every 8 connection(s)
I20260926 11:35:32.870780 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:32.871191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 3521
I20260926 11:35:32.871340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:32.871765 392 external_mini_cluster.cc:1428] Setting key 7500126f010a22e4cda2f1088d901d97
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 26 11:35:32 dist-test-slave-4pw5 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:32.892359 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:32.872925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40271 (local address 127.0.98.62:37995)
0926 11:35:32.873077 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:32.873083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:32.874154 (+ 1071us) server_negotiation.cc:407] Connection header received
0926 11:35:32.876116 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:32.876120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:32.876178 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:32.876269 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:32.878147 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:32.878780 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:32.880846 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:32.881091 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:32.881214 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:32.887625 (+ 6411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:32.887637 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:32.887639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:32.887673 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:32.889244 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:32.890113 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:32.890116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:32.890117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:32.890245 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:32.890787 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:32.890790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:32.890791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:32.891152 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:32.891268 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:32.892079 (+ 811us) server_negotiation.cc:299] Negotiation successful
0926 11:35:32.892226 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:35:32.893713 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:32.873439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37995 (local address 127.0.98.1:40271)
0926 11:35:32.874006 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:32.874053 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:32.875792 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:32.876518 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:32.876534 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:32.877149 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:32.877917 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:32.877943 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:32.878940 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:32.878947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:32.880664 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:32.880679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:32.881290 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:32.881298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:32.881559 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:32.882369 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:32.882397 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:32.887324 (+ 4927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:32.889416 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:32.889427 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:32.889453 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:32.889926 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:32.890358 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:32.890364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:32.890369 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:32.890609 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:32.891301 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:32.891311 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:32.891758 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 11:35:32.892012 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 11:35:32.892497 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":156,"threads_started":1}
I20260926 11:35:32.895223 3650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37995
I20260926 11:35:32.895509 3650 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:32.896148 3650 heartbeater.cc:511] Master 127.0.98.62:37995 requested a full tablet report, sending...
I20260926 11:35:32.897871 3458 ts_manager.cc:194] Registered new tserver with Master: 7ef180748fff408d8ee5141e13d775b5 (127.0.98.1:41011)
I20260926 11:35:32.899084 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:35:32.907646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:33.028119 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:33.028554 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:33.028674 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:33.033077 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:33.033182 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:33.033288 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:33.039664 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:33.040823 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:33.041865 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:33.052326 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:33.052326 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:33.052366 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:33.052568 3655 server_base.cc:1034] running on GCE node
I20260926 11:35:33.053241 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:33.054003 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:33.055248 3655 hybrid_clock.cc:648] HybridClock initialized: now 1790422533055216 us; error 52 us; skew 500 ppm
Sep 26 11:35:33 dist-test-slave-4pw5 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:33.058414 3655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:35:33.059429 3655 webserver.cc:460] Webserver started at http://127.0.98.2:43101/ using document root <none> and password file <none>
I20260926 11:35:33.060101 3655 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:33.060209 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:33.060451 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:33.062598 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "26dd8859dbd8430b80465b5e2dd9a93e"
format_stamp: "Formatted at 2026-09-26 11:35:33 on dist-test-slave-4pw5"
server_key: "5adb79bdb7dc42dd60af30819c73b049"
server_key_iv: "9485c8477f4a8b4200d170e106ee77fd"
server_key_version: "encryptionkey@0"
I20260926 11:35:33.063197 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "26dd8859dbd8430b80465b5e2dd9a93e"
format_stamp: "Formatted at 2026-09-26 11:35:33 on dist-test-slave-4pw5"
server_key: "5adb79bdb7dc42dd60af30819c73b049"
server_key_iv: "9485c8477f4a8b4200d170e106ee77fd"
server_key_version: "encryptionkey@0"
I20260926 11:35:33.067399 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 11:35:33.070708 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:33.072715 3655 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:35:33.072894 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "26dd8859dbd8430b80465b5e2dd9a93e"
format_stamp: "Formatted at 2026-09-26 11:35:33 on dist-test-slave-4pw5"
server_key: "5adb79bdb7dc42dd60af30819c73b049"
server_key_iv: "9485c8477f4a8b4200d170e106ee77fd"
server_key_version: "encryptionkey@0"
I20260926 11:35:33.073127 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:33.086431 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:33.090020 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:33.090262 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:33.090915 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:33.091917 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:33.091967 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:33.092032 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:33.092074 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:33.101961 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46771
I20260926 11:35:33.101982 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46771 every 8 connection(s)
I20260926 11:35:33.102968 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:33.104676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 3655
I20260926 11:35:33.104825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:33.105178 392 external_mini_cluster.cc:1428] Setting key 70f153979df668f74a851aabb6599a63
Sep 26 11:35:33 dist-test-slave-4pw5 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:35:33.121414 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:33.105222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36191 (local address 127.0.98.62:37995)
0926 11:35:33.105367 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:33.105372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:33.106386 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 11:35:33.108480 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:33.108483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:33.108533 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:33.108610 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:33.109970 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.110992 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.112498 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.112945 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.113089 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:33.116157 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:33.116161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:33.116163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:33.116192 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:33.117980 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:33.118745 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:33.118749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:33.118751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:33.118854 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:33.120060 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:33.120064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:33.120065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:33.120347 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:33.120472 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:33.121110 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 11:35:33.121249 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:35:33.122332 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:33.105654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37995 (local address 127.0.98.2:36191)
0926 11:35:33.106196 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:33.106251 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:33.108205 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:33.108859 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:33.108871 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:33.109321 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:33.109825 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:33.109839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.111175 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.111178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:33.112359 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:33.112369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.113143 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.113147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:33.113307 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:33.114000 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:33.114028 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:33.115953 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:33.118159 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:33.118165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:33.118182 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:33.118595 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:33.119657 (+ 1062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:33.119664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:33.119669 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:33.119915 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:33.120508 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:33.120518 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:33.120828 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 11:35:33.121036 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 11:35:33.121410 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":156,"threads_started":1}
I20260926 11:35:33.123612 3784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37995
I20260926 11:35:33.123896 3784 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:33.124462 3784 heartbeater.cc:511] Master 127.0.98.62:37995 requested a full tablet report, sending...
I20260926 11:35:33.125610 3458 ts_manager.cc:194] Registered new tserver with Master: 26dd8859dbd8430b80465b5e2dd9a93e (127.0.98.2:46771)
I20260926 11:35:33.126545 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:35:33.138904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:33.267062 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:33.267393 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:33.267441 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:33.271704 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:33.271780 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:33.271844 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:33.277846 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:33.279301 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:33.280562 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:33.291910 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:33.291910 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:33.292109 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:33.292287 3789 server_base.cc:1034] running on GCE node
I20260926 11:35:33.292840 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:33.293540 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:33.294747 3789 hybrid_clock.cc:648] HybridClock initialized: now 1790422533294728 us; error 39 us; skew 500 ppm
Sep 26 11:35:33 dist-test-slave-4pw5 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:33.298012 3789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:35:33.299074 3789 webserver.cc:460] Webserver started at http://127.0.98.3:41499/ using document root <none> and password file <none>
I20260926 11:35:33.299970 3789 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:33.300078 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:33.300357 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:33.302574 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "aafffb85862646c5850e1bf80287182d"
format_stamp: "Formatted at 2026-09-26 11:35:33 on dist-test-slave-4pw5"
server_key: "9ff635708553b8f7eb5a5b1d0df5650d"
server_key_iv: "050a271de0e1b5f8057aa6bd0a91b8e5"
server_key_version: "encryptionkey@0"
I20260926 11:35:33.303269 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aafffb85862646c5850e1bf80287182d"
format_stamp: "Formatted at 2026-09-26 11:35:33 on dist-test-slave-4pw5"
server_key: "9ff635708553b8f7eb5a5b1d0df5650d"
server_key_iv: "050a271de0e1b5f8057aa6bd0a91b8e5"
server_key_version: "encryptionkey@0"
I20260926 11:35:33.309559 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 11:35:33.313366 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:33.314898 3789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:35:33.315038 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "aafffb85862646c5850e1bf80287182d"
format_stamp: "Formatted at 2026-09-26 11:35:33 on dist-test-slave-4pw5"
server_key: "9ff635708553b8f7eb5a5b1d0df5650d"
server_key_iv: "050a271de0e1b5f8057aa6bd0a91b8e5"
server_key_version: "encryptionkey@0"
I20260926 11:35:33.315214 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:33.330621 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:33.334304 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:33.334529 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:33.335201 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:33.336160 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:33.336234 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:33.336303 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:33.336346 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:33.345422 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37789
I20260926 11:35:33.345469 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37789 every 8 connection(s)
I20260926 11:35:33.346405 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:33.346871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 3789
I20260926 11:35:33.347012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:33.347357 392 external_mini_cluster.cc:1428] Setting key b5dc1f5aaf7992ddc170713727df4f27
Sep 26 11:35:33 dist-test-slave-4pw5 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:33.363499 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:33.348599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38365 (local address 127.0.98.62:37995)
0926 11:35:33.348755 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:33.348759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:33.350103 (+ 1344us) server_negotiation.cc:407] Connection header received
0926 11:35:33.351707 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:33.351710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:33.351813 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:33.351919 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:33.353321 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.353970 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.355424 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.355817 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.355929 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:33.358976 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:33.358979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:33.358982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:33.359011 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:33.360880 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:33.361674 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:33.361678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:33.361680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:33.361791 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:33.362333 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:33.362335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:33.362337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:33.362531 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:33.362610 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:33.363237 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 11:35:33.363380 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:35:33.364393 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:33.349193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37995 (local address 127.0.98.3:38365)
0926 11:35:33.349925 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:33.349980 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:33.351376 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:33.352126 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:33.352135 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:33.352577 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:33.353125 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:33.353141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.354235 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.354238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:33.355237 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:33.355248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.356064 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.356067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:33.356248 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:33.357012 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:33.357031 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:33.358736 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:33.361024 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:33.361031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:33.361053 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:33.361526 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:33.361978 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:33.361983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:33.361986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:33.362229 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:33.362670 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:33.362676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:33.362959 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 11:35:33.363141 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 11:35:33.363440 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":168,"threads_started":1}
I20260926 11:35:33.365543 3918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37995
I20260926 11:35:33.365775 3918 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:33.366278 3918 heartbeater.cc:511] Master 127.0.98.62:37995 requested a full tablet report, sending...
I20260926 11:35:33.367437 3458 ts_manager.cc:194] Registered new tserver with Master: aafffb85862646c5850e1bf80287182d (127.0.98.3:37789)
I20260926 11:35:33.368359 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38365
I20260926 11:35:33.374271 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 11:35:33 dist-test-slave-4pw5 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422533, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 11:35:33 dist-test-slave-4pw5 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422533, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:33.402251 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:33.393170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.0.98.62:37995)
0926 11:35:33.393385 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:33.393390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:33.393509 (+ 119us) server_negotiation.cc:407] Connection header received
0926 11:35:33.393692 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:33.393695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:33.393741 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:33.393833 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:33.394583 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.395120 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.396064 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:33.396283 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:33.396396 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:33.399001 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:33.399005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:33.399007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:33.399035 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:33.400483 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:33.400925 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:33.400929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:33.400930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:33.401016 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:33.401313 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:33.401317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:33.401318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:33.401556 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:33.401703 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:33.401962 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 11:35:33.402084 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260926 11:35:33.402700 3458 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57812
I20260926 11:35:33.403739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 3521
I20260926 11:35:33.409421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 3655
I20260926 11:35:33.414949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 3789
I20260926 11:35:33.420449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 3427
2026-09-26T11:35:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2276 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:33 dist-test-slave-4pw5 krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:33 dist-test-slave-4pw5 krb5kdc[3930](info): set up 2 sockets
Sep 26 11:35:33 dist-test-slave-4pw5 krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260926 11:35:34.547384 392 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 11:35:34.722219 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.175s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 11:35:34.923441 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.201s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 11:35:35.040426 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.117s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/test-user.keytab.
W20260926 11:35:35.700392 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.660s user 0.000s sys 0.002s
Sep 26 11:35:35 dist-test-slave-4pw5 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:35.811481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:35.952340 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:35.952769 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:35.952821 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:35.958036 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:35.958163 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:35.958185 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:35.958204 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:35.958221 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:35.965046 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:35.966543 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:35.967916 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:35.979600 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:35.979789 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:35.980015 3943 server_base.cc:1034] running on GCE node
W20260926 11:35:35.979919 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:35.981051 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:35.982692 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:35.983903 3943 hybrid_clock.cc:648] HybridClock initialized: now 1790422535983884 us; error 47 us; skew 500 ppm
Sep 26 11:35:35 dist-test-slave-4pw5 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:35.987491 3943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:35.988831 3943 webserver.cc:460] Webserver started at http://127.0.98.62:43343/ using document root <none> and password file <none>
I20260926 11:35:35.989611 3943 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:35.989719 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:35.990017 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:35.992811 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "2dbe22288fe846ce917d4b41cb9350b0"
format_stamp: "Formatted at 2026-09-26 11:35:35 on dist-test-slave-4pw5"
server_key: "01fe14aa4029e1dcbd89456083042796"
server_key_iv: "f5b5665629d3cfb6626b1363282604c5"
server_key_version: "encryptionkey@0"
I20260926 11:35:35.993513 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "2dbe22288fe846ce917d4b41cb9350b0"
format_stamp: "Formatted at 2026-09-26 11:35:35 on dist-test-slave-4pw5"
server_key: "01fe14aa4029e1dcbd89456083042796"
server_key_iv: "f5b5665629d3cfb6626b1363282604c5"
server_key_version: "encryptionkey@0"
I20260926 11:35:35.998509 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:35:36.001941 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.003692 3943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:36.003875 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "2dbe22288fe846ce917d4b41cb9350b0"
format_stamp: "Formatted at 2026-09-26 11:35:35 on dist-test-slave-4pw5"
server_key: "01fe14aa4029e1dcbd89456083042796"
server_key_iv: "f5b5665629d3cfb6626b1363282604c5"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.004035 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:36.031910 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:36.036461 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:36.036720 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:36.045907 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39679
I20260926 11:35:36.045975 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39679 every 8 connection(s)
I20260926 11:35:36.048774 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:36.051854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 3943
I20260926 11:35:36.052011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:36.052452 392 external_mini_cluster.cc:1428] Setting key 2bd43e806a03cbf697a36f4aa92e0dbc
I20260926 11:35:36.053543 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:36.062347 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0: Bootstrap starting.
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:36.065974 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:36.066993 4011 log.cc:824] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:36.070807 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0: No bootstrap required, opened a new log
I20260926 11:35:36.072088 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:36.054484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.0.98.62:39679)
0926 11:35:36.055009 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:36.055032 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:36.055088 (+ 56us) server_negotiation.cc:407] Connection header received
0926 11:35:36.055977 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:36.056001 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:36.056491 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:36.056900 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:36.058041 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.059175 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.060492 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.061119 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.061265 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:36.064948 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:36.064956 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:36.064975 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:36.065022 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:36.067469 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.068136 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.068146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.068156 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.068337 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.068780 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.068784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.068786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.069421 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:36.069646 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:36.070097 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 11:35:36.070555 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":168,"threads_started":1}
I20260926 11:35:36.076133 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbe22288fe846ce917d4b41cb9350b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260926 11:35:36.076470 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:36.076568 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dbe22288fe846ce917d4b41cb9350b0, State: Initialized, Role: FOLLOWER
I20260926 11:35:36.077196 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbe22288fe846ce917d4b41cb9350b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260926 11:35:36.077399 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:36.077476 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:36.077596 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:36.078799 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbe22288fe846ce917d4b41cb9350b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260926 11:35:36.079191 4011 leader_election.cc:302] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dbe22288fe846ce917d4b41cb9350b0; no voters:
I20260926 11:35:36.079595 4011 leader_election.cc:288] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:36.079793 4016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:36.080090 4016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2dbe22288fe846ce917d4b41cb9350b0, State: Running, Role: LEADER
I20260926 11:35:36.080509 4016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbe22288fe846ce917d4b41cb9350b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260926 11:35:36.081261 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:36.082810 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dbe22288fe846ce917d4b41cb9350b0. Latest consensus state: current_term: 1 leader_uuid: "2dbe22288fe846ce917d4b41cb9350b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbe22288fe846ce917d4b41cb9350b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260926 11:35:36.082975 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:36.083235 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dbe22288fe846ce917d4b41cb9350b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbe22288fe846ce917d4b41cb9350b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260926 11:35:36.083333 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:36.083777 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:36.087364 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:36.096030 4025 catalog_manager.cc:1347] Generated new cluster ID: 99821ce85fc14100bba86a6a0f3794ba
I20260926 11:35:36.096155 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:36.128608 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:36.130211 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:36.140503 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dbe22288fe846ce917d4b41cb9350b0: Generated new TSK 0
I20260926 11:35:36.141760 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:36.191359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:36.328310 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:36.328626 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:36.328672 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:36.332896 4037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:36.332981 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:36.333050 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:36.338833 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:36.340124 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:36.341231 4037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:36.352357 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:36.352510 4037 server_base.cc:1034] running on GCE node
W20260926 11:35:36.352334 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:36.352317 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:36.353251 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:36.354027 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:36.355281 4037 hybrid_clock.cc:648] HybridClock initialized: now 1790422536355237 us; error 65 us; skew 500 ppm
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:36.358868 4037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:36.360244 4037 webserver.cc:460] Webserver started at http://127.0.98.1:46879/ using document root <none> and password file <none>
I20260926 11:35:36.360927 4037 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:36.361034 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:36.361275 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:36.363965 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "7efff262feb84d74b9b7ada0c3f253e9"
format_stamp: "Formatted at 2026-09-26 11:35:36 on dist-test-slave-4pw5"
server_key: "1be40eb517508e508266b125049f82d7"
server_key_iv: "8fa604d6ecf7821004d93c0015823cbc"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.364666 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7efff262feb84d74b9b7ada0c3f253e9"
format_stamp: "Formatted at 2026-09-26 11:35:36 on dist-test-slave-4pw5"
server_key: "1be40eb517508e508266b125049f82d7"
server_key_iv: "8fa604d6ecf7821004d93c0015823cbc"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.369086 4037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:36.372291 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.373759 4037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:35:36.373912 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "7efff262feb84d74b9b7ada0c3f253e9"
format_stamp: "Formatted at 2026-09-26 11:35:36 on dist-test-slave-4pw5"
server_key: "1be40eb517508e508266b125049f82d7"
server_key_iv: "8fa604d6ecf7821004d93c0015823cbc"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.374048 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:36.401465 4037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:36.405562 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:36.405813 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:36.406464 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:36.407362 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:36.407436 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.407509 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:36.407557 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.417459 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40095
I20260926 11:35:36.417476 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40095 every 8 connection(s)
I20260926 11:35:36.418440 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:36.420415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 4037
I20260926 11:35:36.420570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:36.420962 392 external_mini_cluster.cc:1428] Setting key 31ce249f3d7aa47aa84c9b0f2eb5a8fd
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:35:36.436127 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:36.420622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47097 (local address 127.0.98.62:39679)
0926 11:35:36.420830 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:36.420836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:36.421897 (+ 1061us) server_negotiation.cc:407] Connection header received
0926 11:35:36.424009 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:36.424012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:36.424069 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:36.424148 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:36.425821 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.426476 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.427921 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.428336 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.428462 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:36.431513 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:36.431516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:36.431518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:36.431545 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:36.433274 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.433858 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.433863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.433865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.433978 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.434409 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.434415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.434418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.434693 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:36.434804 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:36.435851 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0926 11:35:36.435987 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 11:35:36.437433 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:36.421131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:47097)
0926 11:35:36.421724 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:36.421787 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:36.423732 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:36.424406 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:36.424419 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:36.425058 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:36.425634 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:36.425650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.426626 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.426630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:36.427762 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:36.427776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.428524 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.428537 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:36.428733 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:36.429453 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:36.429472 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:36.431295 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:36.433423 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:36.433431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:36.433450 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:36.433749 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:36.434126 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:36.434130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:36.434136 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:36.434307 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:36.434802 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:36.434811 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:36.435279 (+ 468us) client_negotiation.cc:769] Sending connection context
0926 11:35:36.435689 (+ 410us) client_negotiation.cc:240] Negotiation successful
0926 11:35:36.436157 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":149,"threads_started":1}
I20260926 11:35:36.439148 4166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260926 11:35:36.439455 4166 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:36.440146 4166 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260926 11:35:36.442174 3975 ts_manager.cc:194] Registered new tserver with Master: 7efff262feb84d74b9b7ada0c3f253e9 (127.0.98.1:40095)
I20260926 11:35:36.444195 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:35:36.456337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:36.582252 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:36.582558 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:36.582607 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:36.587553 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:36.587864 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:36.588038 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:36.595175 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:36.596473 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:36.597612 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:36.608482 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:36.608461 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:36.608456 4171 server_base.cc:1034] running on GCE node
W20260926 11:35:36.608426 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:36.609082 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:36.609923 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:36.611155 4171 hybrid_clock.cc:648] HybridClock initialized: now 1790422536611140 us; error 31 us; skew 500 ppm
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:36.614391 4171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:35:36.615410 4171 webserver.cc:460] Webserver started at http://127.0.98.2:37637/ using document root <none> and password file <none>
I20260926 11:35:36.616073 4171 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:36.616168 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:36.616403 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:36.618664 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "34cad9b06638459b9d186dd458c1b426"
format_stamp: "Formatted at 2026-09-26 11:35:36 on dist-test-slave-4pw5"
server_key: "f35dda0324be3e4f81212a21260646e7"
server_key_iv: "f28fed2646e9843cc0160b3b977cfa51"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.619297 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "34cad9b06638459b9d186dd458c1b426"
format_stamp: "Formatted at 2026-09-26 11:35:36 on dist-test-slave-4pw5"
server_key: "f35dda0324be3e4f81212a21260646e7"
server_key_iv: "f28fed2646e9843cc0160b3b977cfa51"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.623497 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:35:36.626224 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.627638 4171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 11:35:36.627800 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "34cad9b06638459b9d186dd458c1b426"
format_stamp: "Formatted at 2026-09-26 11:35:36 on dist-test-slave-4pw5"
server_key: "f35dda0324be3e4f81212a21260646e7"
server_key_iv: "f28fed2646e9843cc0160b3b977cfa51"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.627929 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:36.643915 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:36.647435 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:36.647713 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:36.648341 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:36.649206 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:36.649278 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.649348 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:36.649401 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.660602 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44655
I20260926 11:35:36.660621 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44655 every 8 connection(s)
I20260926 11:35:36.661657 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:36.665441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 4171
I20260926 11:35:36.665573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:36.665854 392 external_mini_cluster.cc:1428] Setting key d977f0290e941465ab0b000b0c2c6ccd
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:35:36.679320 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:36.663892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47465 (local address 127.0.98.62:39679)
0926 11:35:36.664050 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:36.664055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:36.665088 (+ 1033us) server_negotiation.cc:407] Connection header received
0926 11:35:36.666906 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:36.666909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:36.666976 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:36.667056 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:36.668459 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.669043 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.670640 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.670875 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.670972 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:36.674148 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:36.674155 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:36.674159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:36.674197 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:36.676359 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.677046 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.677050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.677053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.677164 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.677659 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.677664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.677666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.677977 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:36.678128 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:36.678868 (+ 740us) server_negotiation.cc:299] Negotiation successful
0926 11:35:36.679074 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 11:35:36.680428 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:36.664381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:47465)
0926 11:35:36.664916 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:36.664961 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:36.666499 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:36.667256 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:36.667266 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:36.667776 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:36.668301 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:36.668316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.669388 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.669393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:36.670455 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:36.670470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.671073 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.671079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:36.671351 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:36.672053 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:36.672079 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:36.673868 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:36.676562 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:36.676570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:36.676588 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:36.676905 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:36.677338 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:36.677344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:36.677349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:36.677530 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:36.678151 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:36.678163 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:36.678451 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 11:35:36.678817 (+ 366us) client_negotiation.cc:240] Negotiation successful
0926 11:35:36.679365 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:35:36.681797 4300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260926 11:35:36.682096 4300 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:36.682832 4300 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260926 11:35:36.684373 3975 ts_manager.cc:194] Registered new tserver with Master: 34cad9b06638459b9d186dd458c1b426 (127.0.98.2:44655)
I20260926 11:35:36.685030 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:35:36.701057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:36.818421 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:36.818739 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:36.818786 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:36.823053 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:36.823133 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:36.823196 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:36.828876 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:36.830046 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:36.831135 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:36.842211 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:36.842226 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:36.842226 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:36.842589 4305 server_base.cc:1034] running on GCE node
I20260926 11:35:36.843057 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:36.843871 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:36.845088 4305 hybrid_clock.cc:648] HybridClock initialized: now 1790422536845064 us; error 38 us; skew 500 ppm
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:36.848812 4305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:35:36.849992 4305 webserver.cc:460] Webserver started at http://127.0.98.3:43533/ using document root <none> and password file <none>
I20260926 11:35:36.850651 4305 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:36.850778 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:36.851035 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:36.853345 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "59077795d1914a468f9a2f77572196d4"
format_stamp: "Formatted at 2026-09-26 11:35:36 on dist-test-slave-4pw5"
server_key: "f0fea475187c8f5f09dc497bb480ab6b"
server_key_iv: "d4df7dddf7289ac92b5a079e05e3e1bb"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.853956 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "59077795d1914a468f9a2f77572196d4"
format_stamp: "Formatted at 2026-09-26 11:35:36 on dist-test-slave-4pw5"
server_key: "f0fea475187c8f5f09dc497bb480ab6b"
server_key_iv: "d4df7dddf7289ac92b5a079e05e3e1bb"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.858877 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 11:35:36.861971 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.863766 4305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:36.863935 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "59077795d1914a468f9a2f77572196d4"
format_stamp: "Formatted at 2026-09-26 11:35:36 on dist-test-slave-4pw5"
server_key: "f0fea475187c8f5f09dc497bb480ab6b"
server_key_iv: "d4df7dddf7289ac92b5a079e05e3e1bb"
server_key_version: "encryptionkey@0"
I20260926 11:35:36.864094 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:36.880240 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:36.884586 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:36.884859 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:36.885548 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:36.886482 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:36.886588 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.886684 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:36.886727 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:36.896951 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46633
I20260926 11:35:36.897027 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46633 every 8 connection(s)
I20260926 11:35:36.898138 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:36.898582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 4305
I20260926 11:35:36.898731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:36.899133 392 external_mini_cluster.cc:1428] Setting key dad48e5f3256a57523f663519eaa8141
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:36.914332 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:36.900556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45305 (local address 127.0.98.62:39679)
0926 11:35:36.900689 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:36.900711 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:36.901719 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 11:35:36.903437 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:36.903440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:36.903493 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:36.903643 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:36.905055 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.905602 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.906816 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.907106 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.907224 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:36.910371 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:36.910387 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:36.910389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:36.910415 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:36.912036 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.912756 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.912762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.912764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.912848 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.913216 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.913220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.913222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.913413 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:36.913519 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:36.914052 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 11:35:36.914176 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 11:35:36.915297 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:36.901028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:45305)
0926 11:35:36.901539 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:36.901596 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:36.903183 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:36.903918 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:36.903927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:36.904366 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:36.904890 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:36.904904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.905733 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.905736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:36.906672 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:36.906681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.907301 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.907305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:36.907466 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:36.908297 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:36.908314 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:36.910183 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:36.912183 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:36.912190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:36.912208 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:36.912621 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:36.912951 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:36.912954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:36.912957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:36.913117 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:36.913517 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:36.913524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:36.913805 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 11:35:36.914015 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 11:35:36.914392 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":153,"threads_started":1}
I20260926 11:35:36.916633 4434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260926 11:35:36.916879 4434 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:36.917361 4434 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260926 11:35:36.918605 3975 ts_manager.cc:194] Registered new tserver with Master: 59077795d1914a468f9a2f77572196d4 (127.0.98.3:46633)
I20260926 11:35:36.919301 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45305
I20260926 11:35:36.925879 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:35:36.954875 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:36.943336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.0.98.1:40095)
0926 11:35:36.943759 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:36.943766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:36.943794 (+ 28us) server_negotiation.cc:407] Connection header received
0926 11:35:36.943860 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:36.943866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:36.944079 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:36.944214 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:36.944994 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.945768 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.946796 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.947222 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.947310 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:36.950151 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:36.950163 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:36.950201 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:36.950239 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:36.952716 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.953358 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.953368 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.953371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.953482 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.953786 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.953791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.953793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.954070 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:36.954347 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:36.954518 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 11:35:36.954681 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":88,"threads_started":1}
Sep 26 11:35:36 dist-test-slave-4pw5 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:36.966899 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:36.958070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.0.98.62:39679)
0926 11:35:36.958239 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:36.958242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:36.958257 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:35:36.958408 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:36.958412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:36.958460 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:36.958545 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:36.959205 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.959885 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.960724 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:36.960946 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:36.961079 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:36.963648 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:36.963652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:36.963654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:36.963683 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:36.965161 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.965597 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.965601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.965603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.965693 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:36.965973 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:36.965978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:36.965980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:36.966217 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:36.966359 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:36.966598 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 11:35:36.966718 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 11:35:36.967674 3975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45234
I20260926 11:35:36.968434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 4037
I20260926 11:35:36.974983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 4171
I20260926 11:35:36.980523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 4305
I20260926 11:35:36.986573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 3943
2026-09-26T11:35:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3567 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:37 dist-test-slave-4pw5 krb5kdc[4446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:37 dist-test-slave-4pw5 krb5kdc[4446](info): set up 2 sockets
Sep 26 11:35:37 dist-test-slave-4pw5 krb5kdc[4446](info): commencing operation
krb5kdc: starting...
W20260926 11:35:38.060145 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:38 dist-test-slave-4pw5 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:38.191556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36187
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:38.308199 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:38.308516 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:38.308564 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:38.312943 4459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:38.313084 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:38.313119 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:38.313167 4459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:38.313192 4459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:38.323529 4459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:38.324894 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:38.326049 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:38.335137 4467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:38.335197 4465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:38.335203 4464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:38.335701 4459 server_base.cc:1034] running on GCE node
I20260926 11:35:38.336274 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:38.337428 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:38.338637 4459 hybrid_clock.cc:648] HybridClock initialized: now 1790422538338589 us; error 65 us; skew 500 ppm
Sep 26 11:35:38 dist-test-slave-4pw5 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:38.345301 4459 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:38.346201 4459 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:38.346305 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:38.346555 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:38.348937 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc"
format_stamp: "Formatted at 2026-09-26 11:35:38 on dist-test-slave-4pw5"
server_key: "5dae1a2e5be24961672a0d97d6d26c7b"
server_key_iv: "23f328d5f192498f7f6fd8824eb609b6"
server_key_version: "encryptionkey@0"
I20260926 11:35:38.349535 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc"
format_stamp: "Formatted at 2026-09-26 11:35:38 on dist-test-slave-4pw5"
server_key: "5dae1a2e5be24961672a0d97d6d26c7b"
server_key_iv: "23f328d5f192498f7f6fd8824eb609b6"
server_key_version: "encryptionkey@0"
I20260926 11:35:38.353615 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:35:38.356484 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:38.358011 4459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:38.358213 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc"
format_stamp: "Formatted at 2026-09-26 11:35:38 on dist-test-slave-4pw5"
server_key: "5dae1a2e5be24961672a0d97d6d26c7b"
server_key_iv: "23f328d5f192498f7f6fd8824eb609b6"
server_key_version: "encryptionkey@0"
I20260926 11:35:38.358374 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:38.372287 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:38.375841 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:38.376060 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:38.383273 4459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36187
I20260926 11:35:38.383563 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36187 every 8 connection(s)
I20260926 11:35:38.384058 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:38.387120 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:38.388983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 4459
I20260926 11:35:38.389117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:38.389397 392 external_mini_cluster.cc:1428] Setting key 7784300471c8634b4d0027bdfcf84651
I20260926 11:35:38.394220 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc: Bootstrap starting.
I20260926 11:35:38.397058 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:38.398080 4526 log.cc:824] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:35:38 dist-test-slave-4pw5 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:38.400813 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc: No bootstrap required, opened a new log
I20260926 11:35:38.404681 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36187 } }
I20260926 11:35:38.404973 4526 raft_consensus.cc:374] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:38.405077 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06c927a32ed4ed7b90bd6a9ceba0ebc, State: Initialized, Role: FOLLOWER
I20260926 11:35:38.405323 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:38.390862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.0.98.62:36187)
0926 11:35:38.391363 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:38.391381 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:38.391420 (+ 39us) server_negotiation.cc:407] Connection header received
0926 11:35:38.392188 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:38.392207 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:38.392632 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:38.392949 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:38.393889 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.395191 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.396415 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.396959 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.397097 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:38.399964 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:38.399970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:38.399997 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:38.400029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:38.401875 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:38.402343 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:38.402355 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:38.402359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:38.402472 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:38.402919 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:38.402924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:38.402926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:38.403401 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:38.403666 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:38.404018 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 11:35:38.404355 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":167,"threads_started":1}
I20260926 11:35:38.405622 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36187 } }
I20260926 11:35:38.405769 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:38.405855 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:38.405951 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:38.407186 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36187 } }
I20260926 11:35:38.407618 4526 leader_election.cc:302] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a06c927a32ed4ed7b90bd6a9ceba0ebc; no voters:
I20260926 11:35:38.408010 4526 leader_election.cc:288] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:38.408125 4531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:38.408352 4531 raft_consensus.cc:686] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [term 1 LEADER]: Becoming Leader. State: Replica: a06c927a32ed4ed7b90bd6a9ceba0ebc, State: Running, Role: LEADER
I20260926 11:35:38.408659 4531 consensus_queue.cc:237] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36187 } }
I20260926 11:35:38.409328 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:38.410419 4533 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a06c927a32ed4ed7b90bd6a9ceba0ebc. Latest consensus state: current_term: 1 leader_uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36187 } } }
I20260926 11:35:38.410552 4533 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:38.410851 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06c927a32ed4ed7b90bd6a9ceba0ebc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36187 } } }
I20260926 11:35:38.410926 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:38.410948 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:38.414472 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:38.420171 4540 catalog_manager.cc:1347] Generated new cluster ID: b4bfa88a51ad4a0b8a44eeeba19e69e6
I20260926 11:35:38.420249 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:38.439184 4540 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:38.440299 4540 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:38.446046 4540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc: Generated new TSK 0
I20260926 11:35:38.446954 4540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:38.487304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36187
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:38.606769 4552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:38.607100 4552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:38.607151 4552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:38.611639 4552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:38.611718 4552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:38.611785 4552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:38.617548 4552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:38.619499 4552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:38.620831 4552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:35:38.632272 4552 server_base.cc:1034] running on GCE node
W20260926 11:35:38.632270 4557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:38.632228 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:38.632228 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:38.632933 4552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:38.633736 4552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:38.635015 4552 hybrid_clock.cc:648] HybridClock initialized: now 1790422538634962 us; error 68 us; skew 500 ppm
Sep 26 11:35:38 dist-test-slave-4pw5 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:38.638238 4552 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:38.639134 4552 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:38.639241 4552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:38.639535 4552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:38.641877 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "62bb7415afbd46e4b2a379ff5ef5234e"
format_stamp: "Formatted at 2026-09-26 11:35:38 on dist-test-slave-4pw5"
server_key: "b08a0bb51fb8af77b04eda13f7bf078c"
server_key_iv: "769261033d5ed9f1b38f3f21e8f36c42"
server_key_version: "encryptionkey@0"
I20260926 11:35:38.642611 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "62bb7415afbd46e4b2a379ff5ef5234e"
format_stamp: "Formatted at 2026-09-26 11:35:38 on dist-test-slave-4pw5"
server_key: "b08a0bb51fb8af77b04eda13f7bf078c"
server_key_iv: "769261033d5ed9f1b38f3f21e8f36c42"
server_key_version: "encryptionkey@0"
I20260926 11:35:38.647377 4552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 11:35:38.650326 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:38.651901 4552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:38.652056 4552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "62bb7415afbd46e4b2a379ff5ef5234e"
format_stamp: "Formatted at 2026-09-26 11:35:38 on dist-test-slave-4pw5"
server_key: "b08a0bb51fb8af77b04eda13f7bf078c"
server_key_iv: "769261033d5ed9f1b38f3f21e8f36c42"
server_key_version: "encryptionkey@0"
I20260926 11:35:38.652192 4552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:38.669051 4552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:38.672772 4552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:38.673009 4552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:38.673676 4552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:38.674602 4552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:38.674677 4552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:38.674751 4552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:38.674806 4552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:38.684967 4552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45261
I20260926 11:35:38.684978 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45261 every 8 connection(s)
I20260926 11:35:38.685586 4552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:38.694738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 4552
I20260926 11:35:38.694875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:38.695220 392 external_mini_cluster.cc:1428] Setting key 9aa0219f3592855d9a64f039dd952da6
Sep 26 11:35:38 dist-test-slave-4pw5 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:38.700816 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:38.687435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47155 (local address 127.0.98.62:36187)
0926 11:35:38.687647 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:38.687652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:38.688474 (+ 822us) server_negotiation.cc:407] Connection header received
0926 11:35:38.689906 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:38.689910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:38.689967 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:38.690107 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:38.691327 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.691945 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.693195 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.693434 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.693566 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:38.696613 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:38.696618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:38.696620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:38.696647 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:38.698407 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:38.698933 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:38.698939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:38.698942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:38.699038 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:38.699427 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:38.699431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:38.699434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:38.699723 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:38.699830 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:38.700413 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 11:35:38.700566 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:35:38.701574 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:38.687841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36187 (local address 127.0.98.1:47155)
0926 11:35:38.688314 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:38.688360 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:38.689685 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:38.690286 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:38.690293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:38.690675 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:38.691168 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:38.691182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.692151 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.692159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:38.693061 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:38.693071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.693621 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.693626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:38.693798 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:38.694473 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:38.694498 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:38.696451 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:38.698539 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:38.698545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:38.698559 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:38.698831 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:38.699165 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:38.699168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:38.699170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:38.699324 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:38.699853 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:38.699859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:38.700160 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 11:35:38.700327 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 11:35:38.700606 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":114,"threads_started":1}
I20260926 11:35:38.702912 4680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36187
I20260926 11:35:38.703208 4680 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:38.703835 4680 heartbeater.cc:511] Master 127.0.98.62:36187 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:35:38.706027 4490 ts_manager.cc:194] Registered new tserver with Master: 62bb7415afbd46e4b2a379ff5ef5234e (127.0.98.1:45261)
I20260926 11:35:38.707391 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:35:38.728540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36187
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:38.848978 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:38.849299 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:38.849349 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:38.853915 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:38.853999 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:38.854106 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:38.860380 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:38.861693 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:38.862848 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:38.873199 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:38.873281 4685 server_base.cc:1034] running on GCE node
W20260926 11:35:38.873220 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:38.873220 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:38.873936 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:38.874645 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:38.875808 4685 hybrid_clock.cc:648] HybridClock initialized: now 1790422538875815 us; error 66 us; skew 500 ppm
Sep 26 11:35:38 dist-test-slave-4pw5 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:38.878727 4685 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:35:38.879658 4685 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:38.879762 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:38.880029 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:38.882241 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "1978aa69435141528b0d0b42c7ae746d"
format_stamp: "Formatted at 2026-09-26 11:35:38 on dist-test-slave-4pw5"
server_key: "17c6357a0485d06751a016f956bbc861"
server_key_iv: "b4956fe40710aebf52fda078f1285fb4"
server_key_version: "encryptionkey@0"
I20260926 11:35:38.882861 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1978aa69435141528b0d0b42c7ae746d"
format_stamp: "Formatted at 2026-09-26 11:35:38 on dist-test-slave-4pw5"
server_key: "17c6357a0485d06751a016f956bbc861"
server_key_iv: "b4956fe40710aebf52fda078f1285fb4"
server_key_version: "encryptionkey@0"
I20260926 11:35:38.887015 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:35:38.889842 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:38.891171 4685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:38.891327 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "1978aa69435141528b0d0b42c7ae746d"
format_stamp: "Formatted at 2026-09-26 11:35:38 on dist-test-slave-4pw5"
server_key: "17c6357a0485d06751a016f956bbc861"
server_key_iv: "b4956fe40710aebf52fda078f1285fb4"
server_key_version: "encryptionkey@0"
I20260926 11:35:38.891459 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:38.913987 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:38.917845 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:38.918115 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:38.918799 4685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:38.919831 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:38.919883 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:38.919955 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:38.919996 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:38.930747 4685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33407
I20260926 11:35:38.930764 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33407 every 8 connection(s)
I20260926 11:35:38.931414 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:38.936261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 4685
I20260926 11:35:38.936395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:38.936720 392 external_mini_cluster.cc:1428] Setting key 3dec1f502eaffa4d7b8a3cd37c91e24b
Sep 26 11:35:38 dist-test-slave-4pw5 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:35:38.947965 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:38.933400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41235 (local address 127.0.98.62:36187)
0926 11:35:38.933570 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:38.933575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:38.934480 (+ 905us) server_negotiation.cc:407] Connection header received
0926 11:35:38.936066 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:38.936072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:38.936138 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:38.936245 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:38.937807 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.938389 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.939788 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.940009 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.940094 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:38.943219 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:38.943227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:38.943234 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:38.943275 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:38.945451 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:38.946053 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:38.946061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:38.946064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:38.946163 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:38.946632 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:38.946637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:38.946642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:38.946925 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:38.947062 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:38.947553 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 11:35:38.947729 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:35:38.948797 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:38.933818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36187 (local address 127.0.98.2:41235)
0926 11:35:38.934302 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:38.934348 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:38.935805 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:38.936463 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:38.936477 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:38.937000 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:38.937646 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:38.937663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.938544 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.938547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:38.939654 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:38.939670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:38.940180 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:38.940186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:38.940444 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:38.941186 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:38.941215 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:38.943031 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:38.945601 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:38.945607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:38.945621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:38.945928 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:38.946295 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:38.946301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:38.946306 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:38.946518 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:38.947044 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:38.947049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:38.947298 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 11:35:38.947503 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 11:35:38.947893 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":102,"threads_started":1}
I20260926 11:35:38.950232 4813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36187
I20260926 11:35:38.950579 4813 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:35:38.951228 4813 heartbeater.cc:511] Master 127.0.98.62:36187 requested a full tablet report, sending...
I20260926 11:35:38.952556 4490 ts_manager.cc:194] Registered new tserver with Master: 1978aa69435141528b0d0b42c7ae746d (127.0.98.2:33407)
I20260926 11:35:38.953208 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:35:38.970752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36187
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:39.091203 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:39.091578 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:39.091693 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:39.096077 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:39.096184 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:39.096285 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:39.102162 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:39.103361 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:39.104489 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:39.114589 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:39.114534 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:39.114564 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:39.115235 4818 server_base.cc:1034] running on GCE node
I20260926 11:35:39.115780 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:39.116422 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:39.117631 4818 hybrid_clock.cc:648] HybridClock initialized: now 1790422539117600 us; error 48 us; skew 500 ppm
Sep 26 11:35:39 dist-test-slave-4pw5 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:39.120851 4818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:35:39.121693 4818 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:39.121764 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:39.122013 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:39.124517 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e776ccfd7404f80b24331da44d8df87"
format_stamp: "Formatted at 2026-09-26 11:35:39 on dist-test-slave-4pw5"
server_key: "f8e4cff639f11f6b61bf09ce961ff6d3"
server_key_iv: "15996af14a3b8d0c6c8d81343588dd19"
server_key_version: "encryptionkey@0"
I20260926 11:35:39.125154 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e776ccfd7404f80b24331da44d8df87"
format_stamp: "Formatted at 2026-09-26 11:35:39 on dist-test-slave-4pw5"
server_key: "f8e4cff639f11f6b61bf09ce961ff6d3"
server_key_iv: "15996af14a3b8d0c6c8d81343588dd19"
server_key_version: "encryptionkey@0"
I20260926 11:35:39.129384 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 11:35:39.132153 4832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:39.133628 4818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:39.133821 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "2e776ccfd7404f80b24331da44d8df87"
format_stamp: "Formatted at 2026-09-26 11:35:39 on dist-test-slave-4pw5"
server_key: "f8e4cff639f11f6b61bf09ce961ff6d3"
server_key_iv: "15996af14a3b8d0c6c8d81343588dd19"
server_key_version: "encryptionkey@0"
I20260926 11:35:39.133955 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:39.148595 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:39.152271 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:39.152484 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:39.153097 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:39.153957 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:39.154003 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:39.154043 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:39.154089 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:39.163753 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40699
I20260926 11:35:39.163769 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40699 every 8 connection(s)
I20260926 11:35:39.164422 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:39.167832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 4818
I20260926 11:35:39.167985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:39.168274 392 external_mini_cluster.cc:1428] Setting key d2cee5dc13db35414b9523e4bc35dcf9
Sep 26 11:35:39 dist-test-slave-4pw5 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:39.178706 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.166374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45951 (local address 127.0.98.62:36187)
0926 11:35:39.166531 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.166536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.167421 (+ 885us) server_negotiation.cc:407] Connection header received
0926 11:35:39.169087 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.169090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.169145 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.169232 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:39.170565 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.171095 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.172359 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.172592 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.172706 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.175217 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:39.175220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:39.175222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:39.175246 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:39.176612 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:39.177129 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:39.177133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:39.177135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:39.177216 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:39.177595 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:39.177598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:39.177599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:39.177770 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:39.177880 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.178394 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.178538 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:35:39.179526 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.166784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36187 (local address 127.0.98.3:45951)
0926 11:35:39.167265 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.167308 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:39.168839 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:39.169423 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:39.169430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:39.169802 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:39.170416 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.170434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.171242 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.171250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.172232 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.172242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.172745 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.172749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.172927 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.173553 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:39.173570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:39.175024 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:39.176730 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:39.176736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:39.176753 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:39.177017 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:39.177312 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:39.177316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:39.177318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:39.177458 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:39.177877 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:39.177884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:39.178159 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 11:35:39.178356 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 11:35:39.178687 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":116,"threads_started":1}
I20260926 11:35:39.181001 4946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36187
I20260926 11:35:39.181270 4946 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:39.181845 4946 heartbeater.cc:511] Master 127.0.98.62:36187 requested a full tablet report, sending...
I20260926 11:35:39.183105 4490 ts_manager.cc:194] Registered new tserver with Master: 2e776ccfd7404f80b24331da44d8df87 (127.0.98.3:40699)
I20260926 11:35:39.183801 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45951
I20260926 11:35:39.194391 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:35:39.206185 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.196598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.0.98.62:36187)
0926 11:35:39.196725 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.196729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.197006 (+ 277us) server_negotiation.cc:407] Connection header received
0926 11:35:39.197185 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.197188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.197265 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.197367 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:39.198305 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.198981 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.200332 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.200768 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.200870 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.202834 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:39.202838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:39.202840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:39.202893 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:39.204400 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:39.204854 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:39.204856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:39.204858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:39.204928 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:39.205290 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:39.205293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:39.205294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:39.205583 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:39.205705 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.205947 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.206059 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:35:39.209319 4490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:39.212920 4490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:39.229727 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.222924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33407 (local address 127.0.0.1:46900)
0926 11:35:39.224162 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.224175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:39.224275 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:39.225095 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:39.225097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:39.225115 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:39.225260 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.225269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.227038 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.227042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.227987 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.227997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.229077 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.229081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.229209 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.229258 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 11:35:39.229385 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 11:35:39.229564 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":39,"thread_start_us":117,"threads_started":1}
I20260926 11:35:39.230238 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.222931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.0.98.1:45261)
0926 11:35:39.223742 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.223750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.223775 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:35:39.224483 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.224489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.224694 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.224880 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:39.225209 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.226366 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.228576 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.229373 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.229477 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.229620 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.229956 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.230069 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":111,"threads_started":1}
I20260926 11:35:39.230376 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.223543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.0.98.2:33407)
0926 11:35:39.223989 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.224017 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.224299 (+ 282us) server_negotiation.cc:407] Connection header received
0926 11:35:39.224446 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.224453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.224694 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.224898 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:39.225411 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.226904 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.228153 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.228891 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.229009 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.229185 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.230070 (+ 885us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.230179 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":105,"threads_started":1}
I20260926 11:35:39.230746 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.223451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40699 (local address 127.0.0.1:59480)
0926 11:35:39.224469 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.224478 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:39.224586 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:39.225234 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:39.225236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:39.225260 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:39.225322 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.225325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.226999 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.227004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.228427 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.228435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.230404 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.230408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.230494 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.230517 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:35:39.230588 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 11:35:39.230643 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":104,"thread_start_us":86,"threads_started":1}
I20260926 11:35:39.230747 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.222625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45261 (local address 127.0.0.1:34736)
0926 11:35:39.223466 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.223608 (+ 142us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:39.224224 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:39.224868 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:39.224874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:39.224894 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:39.225079 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.225087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.226550 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.226553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.228422 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.228435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.229686 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.229689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.229788 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.229810 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:35:39.230494 (+ 684us) client_negotiation.cc:240] Negotiation successful
0926 11:35:39.230588 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":30,"thread_start_us":111,"threads_started":1}
I20260926 11:35:39.231242 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.223821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.0.98.3:40699)
0926 11:35:39.224266 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.224283 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.224568 (+ 285us) server_negotiation.cc:407] Connection header received
0926 11:35:39.224718 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.224723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.224934 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.225077 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:39.225465 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.226832 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.229271 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.230079 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.230198 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.230334 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.230987 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.231093 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":175,"threads_started":1}
I20260926 11:35:39.233748 4880 tablet_service.cc:1467] Processing CreateTablet for tablet 80fc4acad7a545368979afc0b04adb27 (DEFAULT_TABLE table=test-table [id=f33f439c485f444fae4a1452d660e1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:39.233748 4614 tablet_service.cc:1467] Processing CreateTablet for tablet 80fc4acad7a545368979afc0b04adb27 (DEFAULT_TABLE table=test-table [id=f33f439c485f444fae4a1452d660e1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:39.233749 4747 tablet_service.cc:1467] Processing CreateTablet for tablet 80fc4acad7a545368979afc0b04adb27 (DEFAULT_TABLE table=test-table [id=f33f439c485f444fae4a1452d660e1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:39.235128 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fc4acad7a545368979afc0b04adb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:39.235163 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fc4acad7a545368979afc0b04adb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:39.235163 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fc4acad7a545368979afc0b04adb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:39.242173 4963 tablet_bootstrap.cc:492] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: Bootstrap starting.
I20260926 11:35:39.243818 4965 tablet_bootstrap.cc:492] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: Bootstrap starting.
I20260926 11:35:39.243950 4964 tablet_bootstrap.cc:492] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e: Bootstrap starting.
I20260926 11:35:39.245240 4963 tablet_bootstrap.cc:654] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:39.246479 4964 tablet_bootstrap.cc:654] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:39.246479 4965 tablet_bootstrap.cc:654] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:39.246613 4963 log.cc:824] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:39.247534 4964 log.cc:824] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:39.247910 4965 log.cc:824] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:39.248953 4963 tablet_bootstrap.cc:492] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: No bootstrap required, opened a new log
I20260926 11:35:39.249183 4963 ts_tablet_manager.cc:1397] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 11:35:39.249828 4965 tablet_bootstrap.cc:492] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: No bootstrap required, opened a new log
I20260926 11:35:39.249827 4964 tablet_bootstrap.cc:492] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e: No bootstrap required, opened a new log
I20260926 11:35:39.250044 4964 ts_tablet_manager.cc:1397] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 11:35:39.250044 4965 ts_tablet_manager.cc:1397] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:35:39.252835 4963 raft_consensus.cc:348] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.253189 4963 raft_consensus.cc:374] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:39.253250 4963 raft_consensus.cc:729] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e776ccfd7404f80b24331da44d8df87, State: Initialized, Role: FOLLOWER
I20260926 11:35:39.253736 4963 consensus_queue.cc:260] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.254526 4946 heartbeater.cc:503] Master 127.0.98.62:36187 was elected leader, sending a full tablet report...
I20260926 11:35:39.254886 4963 ts_tablet_manager.cc:1428] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:35:39.255051 4964 raft_consensus.cc:348] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.255364 4964 raft_consensus.cc:374] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:39.255438 4964 raft_consensus.cc:729] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bb7415afbd46e4b2a379ff5ef5234e, State: Initialized, Role: FOLLOWER
I20260926 11:35:39.255363 4965 raft_consensus.cc:348] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.255718 4965 raft_consensus.cc:374] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:39.255790 4965 raft_consensus.cc:729] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1978aa69435141528b0d0b42c7ae746d, State: Initialized, Role: FOLLOWER
I20260926 11:35:39.255937 4964 consensus_queue.cc:260] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.256242 4965 consensus_queue.cc:260] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.256780 4680 heartbeater.cc:503] Master 127.0.98.62:36187 was elected leader, sending a full tablet report...
I20260926 11:35:39.257078 4964 ts_tablet_manager.cc:1428] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260926 11:35:39.257221 4965 ts_tablet_manager.cc:1428] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 11:35:39.257575 4813 heartbeater.cc:503] Master 127.0.98.62:36187 was elected leader, sending a full tablet report...
I20260926 11:35:39.352787 4971 raft_consensus.cc:482] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:39.352859 4969 raft_consensus.cc:482] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:39.353032 4971 raft_consensus.cc:504] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.353071 4969 raft_consensus.cc:504] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.354250 4971 leader_election.cc:288] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62bb7415afbd46e4b2a379ff5ef5234e (127.0.98.1:45261), 2e776ccfd7404f80b24331da44d8df87 (127.0.98.3:40699)
I20260926 11:35:39.354262 4969 leader_election.cc:288] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62bb7415afbd46e4b2a379ff5ef5234e (127.0.98.1:45261), 1978aa69435141528b0d0b42c7ae746d (127.0.98.2:33407)
I20260926 11:35:39.359793 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.354497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45261 (local address 127.0.98.2:55173)
0926 11:35:39.354934 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.354949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:39.355023 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:39.355994 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:39.355996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:39.356026 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:39.356145 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.356153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.357535 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.357540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.358329 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.358338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.359288 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.359292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.359397 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.359435 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 11:35:39.359486 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 11:35:39.359597 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":11,"spinlock_wait_cycles":1408}
I20260926 11:35:39.359947 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.355520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55459 (local address 127.0.98.3:40699)
0926 11:35:39.355649 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.355654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.355667 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:35:39.355709 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.355712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.355754 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.355830 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:39.356277 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.357182 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.358763 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.359319 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.359423 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.359500 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.359738 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.359815 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":66}
I20260926 11:35:39.360093 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.355064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37511 (local address 127.0.98.1:45261)
0926 11:35:39.355192 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.355197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.355213 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:35:39.355272 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.355276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.355341 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.355419 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:39.355778 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.356721 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.358436 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.359147 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.359259 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.359321 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.359876 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.359949 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":52}
I20260926 11:35:39.360298 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.354769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40699 (local address 127.0.98.2:55459)
0926 11:35:39.355272 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.355285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:39.355374 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:39.355942 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:39.355948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:39.355971 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:39.356152 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.356158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.357309 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.357312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.358614 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.358626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.359480 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.359484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.359606 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.359625 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 11:35:39.360073 (+ 448us) client_negotiation.cc:240] Negotiation successful
0926 11:35:39.360153 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":37,"spinlock_wait_cycles":12928,"thread_start_us":87,"threads_started":1}
I20260926 11:35:39.360395 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.354496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45261 (local address 127.0.98.3:37511)
0926 11:35:39.354649 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.354666 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:39.354770 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:39.355440 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:39.355444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:39.355463 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:39.355645 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.355652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.356868 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.356871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.358271 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.358283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.359380 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.359384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.359643 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.359663 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:35:39.359732 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 11:35:39.359814 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":168}
I20260926 11:35:39.360778 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.354736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36827 (local address 127.0.98.2:33407)
0926 11:35:39.354852 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.354859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.355161 (+ 302us) server_negotiation.cc:407] Connection header received
0926 11:35:39.355345 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.355348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.355410 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.355459 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:39.355887 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.356721 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.359545 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.360108 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.360188 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.360240 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.360598 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.360668 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":3328}
I20260926 11:35:39.360849 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.355200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55173 (local address 127.0.98.1:45261)
0926 11:35:39.355771 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.355783 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.355798 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:35:39.355840 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.355842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.355879 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.356293 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:39.356516 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.357348 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.358491 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.359160 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.360546 (+ 1386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.360604 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.360637 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.360703 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":506,"thread_start_us":107,"threads_started":1}
I20260926 11:35:39.361009 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fc4acad7a545368979afc0b04adb27" candidate_uuid: "1978aa69435141528b0d0b42c7ae746d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e776ccfd7404f80b24331da44d8df87" is_pre_election: true
I20260926 11:35:39.361019 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fc4acad7a545368979afc0b04adb27" candidate_uuid: "2e776ccfd7404f80b24331da44d8df87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" is_pre_election: true
I20260926 11:35:39.361157 4633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fc4acad7a545368979afc0b04adb27" candidate_uuid: "1978aa69435141528b0d0b42c7ae746d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" is_pre_election: true
I20260926 11:35:39.361280 4900 raft_consensus.cc:2454] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1978aa69435141528b0d0b42c7ae746d in term 0.
I20260926 11:35:39.361353 4634 raft_consensus.cc:2454] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e776ccfd7404f80b24331da44d8df87 in term 0.
I20260926 11:35:39.361739 4702 leader_election.cc:302] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1978aa69435141528b0d0b42c7ae746d, 2e776ccfd7404f80b24331da44d8df87; no voters:
I20260926 11:35:39.361815 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.354622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33407 (local address 127.0.98.3:36827)
0926 11:35:39.355082 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.355096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:39.355223 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:39.355640 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:39.355643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:39.355658 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:39.355775 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.355781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.356868 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.356871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.359422 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:39.359429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.360332 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.360336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:39.360436 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.360464 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 11:35:39.361502 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0926 11:35:39.361680 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":49,"thread_start_us":140,"threads_started":1}
I20260926 11:35:39.361806 4835 leader_election.cc:302] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e776ccfd7404f80b24331da44d8df87, 62bb7415afbd46e4b2a379ff5ef5234e; no voters:
I20260926 11:35:39.362090 4969 raft_consensus.cc:2781] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:39.362114 4971 raft_consensus.cc:2781] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:39.362176 4969 raft_consensus.cc:482] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:39.362186 4971 raft_consensus.cc:482] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:39.362228 4969 raft_consensus.cc:3037] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:39.362234 4971 raft_consensus.cc:3037] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:39.362365 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fc4acad7a545368979afc0b04adb27" candidate_uuid: "2e776ccfd7404f80b24331da44d8df87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978aa69435141528b0d0b42c7ae746d" is_pre_election: true
I20260926 11:35:39.363529 4969 raft_consensus.cc:504] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.363529 4971 raft_consensus.cc:504] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.363828 4767 raft_consensus.cc:2379] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e776ccfd7404f80b24331da44d8df87 in current term 1: Already voted for candidate 1978aa69435141528b0d0b42c7ae746d in this term.
I20260926 11:35:39.363932 4969 leader_election.cc:288] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [CANDIDATE]: Term 1 election: Requested vote from peers 62bb7415afbd46e4b2a379ff5ef5234e (127.0.98.1:45261), 1978aa69435141528b0d0b42c7ae746d (127.0.98.2:33407)
I20260926 11:35:39.364287 4633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fc4acad7a545368979afc0b04adb27" candidate_uuid: "1978aa69435141528b0d0b42c7ae746d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bb7415afbd46e4b2a379ff5ef5234e"
I20260926 11:35:39.364338 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fc4acad7a545368979afc0b04adb27" candidate_uuid: "2e776ccfd7404f80b24331da44d8df87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bb7415afbd46e4b2a379ff5ef5234e"
I20260926 11:35:39.364413 4633 raft_consensus.cc:3037] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:39.364733 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fc4acad7a545368979afc0b04adb27" candidate_uuid: "1978aa69435141528b0d0b42c7ae746d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e776ccfd7404f80b24331da44d8df87"
I20260926 11:35:39.364879 4900 raft_consensus.cc:2379] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1978aa69435141528b0d0b42c7ae746d in current term 1: Already voted for candidate 2e776ccfd7404f80b24331da44d8df87 in this term.
I20260926 11:35:39.364893 4766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fc4acad7a545368979afc0b04adb27" candidate_uuid: "2e776ccfd7404f80b24331da44d8df87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978aa69435141528b0d0b42c7ae746d"
I20260926 11:35:39.365036 4766 raft_consensus.cc:2379] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2e776ccfd7404f80b24331da44d8df87 in current term 1: Already voted for candidate 1978aa69435141528b0d0b42c7ae746d in this term.
I20260926 11:35:39.364194 4971 leader_election.cc:288] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [CANDIDATE]: Term 1 election: Requested vote from peers 62bb7415afbd46e4b2a379ff5ef5234e (127.0.98.1:45261), 2e776ccfd7404f80b24331da44d8df87 (127.0.98.3:40699)
I20260926 11:35:39.365365 4833 leader_election.cc:302] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e776ccfd7404f80b24331da44d8df87; no voters: 1978aa69435141528b0d0b42c7ae746d, 62bb7415afbd46e4b2a379ff5ef5234e
I20260926 11:35:39.365574 4969 raft_consensus.cc:2726] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 11:35:39.365720 4633 raft_consensus.cc:2454] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1978aa69435141528b0d0b42c7ae746d in term 1.
I20260926 11:35:39.366019 4702 leader_election.cc:302] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1978aa69435141528b0d0b42c7ae746d, 62bb7415afbd46e4b2a379ff5ef5234e; no voters: 2e776ccfd7404f80b24331da44d8df87
I20260926 11:35:39.366194 4971 raft_consensus.cc:2781] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:39.366415 4971 raft_consensus.cc:686] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 1 LEADER]: Becoming Leader. State: Replica: 1978aa69435141528b0d0b42c7ae746d, State: Running, Role: LEADER
I20260926 11:35:39.366744 4971 consensus_queue.cc:237] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } }
I20260926 11:35:39.369953 4490 catalog_manager.cc:5495] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1978aa69435141528b0d0b42c7ae746d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1978aa69435141528b0d0b42c7ae746d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1978aa69435141528b0d0b42c7ae746d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260926 11:35:39.383113 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.379498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.0.98.2:33407)
0926 11:35:39.379667 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.379670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.379683 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:35:39.379797 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.379799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.379850 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.379940 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:39.380160 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.380667 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.381584 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:39.381805 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:39.381886 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:39.382193 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:39.382658 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:39.382751 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.382837 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.382890 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 11:35:39.389801 4900 raft_consensus.cc:1261] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 1 FOLLOWER]: Refusing update from remote peer 1978aa69435141528b0d0b42c7ae746d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:39.389856 4633 raft_consensus.cc:1261] T 80fc4acad7a545368979afc0b04adb27 P 62bb7415afbd46e4b2a379ff5ef5234e [term 1 FOLLOWER]: Refusing update from remote peer 1978aa69435141528b0d0b42c7ae746d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:39.390753 4971 consensus_queue.cc:1035] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e776ccfd7404f80b24331da44d8df87" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:39.391022 4975 consensus_queue.cc:1035] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [LEADER]: Connected to new peer: Peer: permanent_uuid: "62bb7415afbd46e4b2a379ff5ef5234e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:39.404507 4980 mvcc.cc:205] Tried to move back new op lower bound from 7333570721333030912 to 7333570721249013760. Current Snapshot: MvccSnapshot[applied={T|T < 7333570721333030912}]
I20260926 11:35:39.408489 4489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:39.408653 4489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:39.411341 4489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a06c927a32ed4ed7b90bd6a9ceba0ebc: Sending DeleteTablet for 3 replicas of tablet 80fc4acad7a545368979afc0b04adb27
I20260926 11:35:39.412331 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fc4acad7a545368979afc0b04adb27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46900
I20260926 11:35:39.412549 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fc4acad7a545368979afc0b04adb27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34736
I20260926 11:35:39.412910 4988 tablet_replica.cc:330] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: stopping tablet replica
I20260926 11:35:39.412855 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fc4acad7a545368979afc0b04adb27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59480
I20260926 11:35:39.413280 4988 raft_consensus.cc:2229] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:39.413359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 4552
I20260926 11:35:39.414463 4988 raft_consensus.cc:2258] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:39.416182 4988 ts_tablet_manager.cc:1905] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:39.419745 4988 ts_tablet_manager.cc:1918] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:39.419857 4988 log.cc:1196] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/80fc4acad7a545368979afc0b04adb27
I20260926 11:35:39.419991 4990 tablet_replica.cc:330] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: stopping tablet replica
I20260926 11:35:39.420197 4988 ts_tablet_manager.cc:1939] T 80fc4acad7a545368979afc0b04adb27 P 1978aa69435141528b0d0b42c7ae746d: Deleting consensus metadata
I20260926 11:35:39.420418 4990 raft_consensus.cc:2229] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:39.420804 4990 raft_consensus.cc:2258] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:39.421545 4474 catalog_manager.cc:4841] TS 1978aa69435141528b0d0b42c7ae746d (127.0.98.2:33407): tablet 80fc4acad7a545368979afc0b04adb27 (table test-table [id=f33f439c485f444fae4a1452d660e1a8]) successfully deleted
I20260926 11:35:39.422449 4990 ts_tablet_manager.cc:1905] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 11:35:39.422868 4476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45261 (error 108)
I20260926 11:35:39.423110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 4685
I20260926 11:35:39.425086 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.424669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45261 (local address 127.0.0.1:34744)
0926 11:35:39.424879 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.424972 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260926 11:35:39.425280 4476 catalog_manager.cc:4584] TS 62bb7415afbd46e4b2a379ff5ef5234e (127.0.98.1:45261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80fc4acad7a545368979afc0b04adb27: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45261: connect: Connection refused (error 111)
I20260926 11:35:39.427116 4990 ts_tablet_manager.cc:1918] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:39.427243 4990 log.cc:1196] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/80fc4acad7a545368979afc0b04adb27
I20260926 11:35:39.427627 4990 ts_tablet_manager.cc:1939] T 80fc4acad7a545368979afc0b04adb27 P 2e776ccfd7404f80b24331da44d8df87: Deleting consensus metadata
I20260926 11:35:39.429167 4476 catalog_manager.cc:4841] TS 2e776ccfd7404f80b24331da44d8df87 (127.0.98.3:40699): tablet 80fc4acad7a545368979afc0b04adb27 (table test-table [id=f33f439c485f444fae4a1452d660e1a8]) successfully deleted
I20260926 11:35:39.430725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 4818
I20260926 11:35:39.436762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 4459
2026-09-26T11:35:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2449 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T11:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:39Z Disabled control of system clock
I20260926 11:35:39.468833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34677
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 11:35:39.584834 4995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:39.585147 4995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:39.585189 4995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:39.589697 4995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:39.589788 4995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:39.589804 4995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:39.589823 4995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:39.589838 4995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:39.596213 4995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:39.597940 4995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:39.599627 4995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:39.609189 5001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:39.609189 5003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:39.609205 5000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:39.609489 4995 server_base.cc:1034] running on GCE node
I20260926 11:35:39.610173 4995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:39.611353 4995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:39.612536 4995 hybrid_clock.cc:648] HybridClock initialized: now 1790422539612525 us; error 44 us; skew 500 ppm
I20260926 11:35:39.614748 4995 webserver.cc:460] Webserver started at http://127.0.98.62:36273/ using document root <none> and password file <none>
I20260926 11:35:39.615511 4995 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:39.615650 4995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:39.615938 4995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:39.618396 4995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "fc93a3d909ba412a984b1c3a2db593f4"
format_stamp: "Formatted at 2026-09-26 11:35:39 on dist-test-slave-4pw5"
server_key: "5fae9a46a1ce23bd168e7ff758dabb02"
server_key_iv: "ba2a01ec4e8f7de469a59e7052204ea3"
server_key_version: "encryptionkey@0"
I20260926 11:35:39.619079 4995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc93a3d909ba412a984b1c3a2db593f4"
format_stamp: "Formatted at 2026-09-26 11:35:39 on dist-test-slave-4pw5"
server_key: "5fae9a46a1ce23bd168e7ff758dabb02"
server_key_iv: "ba2a01ec4e8f7de469a59e7052204ea3"
server_key_version: "encryptionkey@0"
I20260926 11:35:39.623267 4995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 11:35:39.625964 5009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:39.627302 4995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 11:35:39.627457 4995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "fc93a3d909ba412a984b1c3a2db593f4"
format_stamp: "Formatted at 2026-09-26 11:35:39 on dist-test-slave-4pw5"
server_key: "5fae9a46a1ce23bd168e7ff758dabb02"
server_key_iv: "ba2a01ec4e8f7de469a59e7052204ea3"
server_key_version: "encryptionkey@0"
I20260926 11:35:39.627619 4995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:39.632416 4995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:39.633068 4995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:39.633256 4995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:39.640591 4995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34677
I20260926 11:35:39.640602 5061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34677 every 8 connection(s)
I20260926 11:35:39.641601 4995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:39.644946 5062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:39.645407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 4995
I20260926 11:35:39.645572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:39.645857 392 external_mini_cluster.cc:1428] Setting key 7584b06c8be409973ca455dd72f09128
I20260926 11:35:39.652174 5062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4: Bootstrap starting.
I20260926 11:35:39.652225 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.647313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.0.98.62:34677)
0926 11:35:39.648091 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.648115 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.648161 (+ 46us) server_negotiation.cc:407] Connection header received
0926 11:35:39.649005 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.649029 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.649488 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.649871 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:39.650665 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:39.650671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:39.650687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:39.650699 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:39.650766 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:39.650791 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:39.650952 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.651023 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.651276 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":120,"threads_started":1}
I20260926 11:35:39.654764 5062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:39.655478 5062 log.cc:824] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:39.657567 5062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4: No bootstrap required, opened a new log
I20260926 11:35:39.661310 5062 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc93a3d909ba412a984b1c3a2db593f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34677 } }
I20260926 11:35:39.661553 5062 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:39.661588 5062 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc93a3d909ba412a984b1c3a2db593f4, State: Initialized, Role: FOLLOWER
I20260926 11:35:39.662014 5062 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc93a3d909ba412a984b1c3a2db593f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34677 } }
I20260926 11:35:39.662146 5062 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:39.662189 5062 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:39.662257 5062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:39.663232 5062 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc93a3d909ba412a984b1c3a2db593f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34677 } }
I20260926 11:35:39.663695 5062 leader_election.cc:302] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc93a3d909ba412a984b1c3a2db593f4; no voters:
I20260926 11:35:39.664055 5062 leader_election.cc:288] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:39.664178 5067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:39.664443 5067 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [term 1 LEADER]: Becoming Leader. State: Replica: fc93a3d909ba412a984b1c3a2db593f4, State: Running, Role: LEADER
I20260926 11:35:39.664790 5067 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc93a3d909ba412a984b1c3a2db593f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34677 } }
I20260926 11:35:39.665382 5062 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:39.666474 5069 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc93a3d909ba412a984b1c3a2db593f4. Latest consensus state: current_term: 1 leader_uuid: "fc93a3d909ba412a984b1c3a2db593f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc93a3d909ba412a984b1c3a2db593f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34677 } } }
I20260926 11:35:39.666610 5069 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:39.666904 5068 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc93a3d909ba412a984b1c3a2db593f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc93a3d909ba412a984b1c3a2db593f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34677 } } }
I20260926 11:35:39.667022 5068 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:39.667114 5076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:39.670584 5076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:39.676604 5076 catalog_manager.cc:1347] Generated new cluster ID: 4dc0ccd2cd194e82bba7ed12a9c96f93
I20260926 11:35:39.676697 5076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:39.705363 5076 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:39.705742 5076 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:39.713357 5076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4: Generated new TSK 0
I20260926 11:35:39.714323 5076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:35:39.722389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34677
--builtin_ntp_servers=127.0.98.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 11:35:39.863538 5086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:39.863955 5086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:39.864048 5086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:39.868642 5086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:39.868744 5086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:39.868860 5086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:39.874145 5086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:39.875712 5086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:39.877256 5086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:39.887698 5094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:39.887874 5086 server_base.cc:1034] running on GCE node
W20260926 11:35:39.887730 5092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:39.888082 5091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:39.888571 5086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:39.889297 5086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:39.890519 5086 hybrid_clock.cc:648] HybridClock initialized: now 1790422539890501 us; error 33 us; skew 500 ppm
I20260926 11:35:39.892683 5086 webserver.cc:460] Webserver started at http://127.0.98.1:39817/ using document root <none> and password file <none>
I20260926 11:35:39.893290 5086 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:39.893355 5086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:39.893579 5086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:39.895938 5086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "7414624177a74d708aea0127584ad9f4"
format_stamp: "Formatted at 2026-09-26 11:35:39 on dist-test-slave-4pw5"
server_key: "dd85068e28fb23e48b5e7c3d1c8196ec"
server_key_iv: "02761e9a2702e8d362a3f06cc2c23bcf"
server_key_version: "encryptionkey@0"
I20260926 11:35:39.896566 5086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7414624177a74d708aea0127584ad9f4"
format_stamp: "Formatted at 2026-09-26 11:35:39 on dist-test-slave-4pw5"
server_key: "dd85068e28fb23e48b5e7c3d1c8196ec"
server_key_iv: "02761e9a2702e8d362a3f06cc2c23bcf"
server_key_version: "encryptionkey@0"
I20260926 11:35:39.900625 5086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 11:35:39.903389 5100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:39.904985 5086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:35:39.905108 5086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "7414624177a74d708aea0127584ad9f4"
format_stamp: "Formatted at 2026-09-26 11:35:39 on dist-test-slave-4pw5"
server_key: "dd85068e28fb23e48b5e7c3d1c8196ec"
server_key_iv: "02761e9a2702e8d362a3f06cc2c23bcf"
server_key_version: "encryptionkey@0"
I20260926 11:35:39.905217 5086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:39.909909 5086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:39.910476 5086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:39.910619 5086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:39.911104 5086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:39.912112 5086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:39.912160 5086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:39.912199 5086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:39.912214 5086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:39.921684 5086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34779
I20260926 11:35:39.921722 5213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34779 every 8 connection(s)
I20260926 11:35:39.922667 5086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:39.929540 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.924866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41385 (local address 127.0.98.62:34677)
0926 11:35:39.925049 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:39.925054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:39.926150 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 11:35:39.927557 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:39.927561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:39.927670 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:39.927753 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:39.929005 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:39.929012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:39.929014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:39.929027 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:39.929049 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:39.929058 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:39.929153 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:39.929263 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 11:35:39.929305 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 11:35:39.929652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 5086
I20260926 11:35:39.929772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:39.930152 392 external_mini_cluster.cc:1428] Setting key f7af2ca402d109cea174561736abbcc6
I20260926 11:35:39.930557 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:39.925243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34677 (local address 127.0.98.1:41385)
0926 11:35:39.925978 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:39.926032 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:39.927306 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:39.927952 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:39.927960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:39.928055 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:39.928706 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:35:39.928707 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:39.928758 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:35:39.928828 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:39.929136 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:39.929141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:39.929144 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 11:35:39.929269 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 11:35:39.929466 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":116,"threads_started":1}
I20260926 11:35:39.932123 5214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34677
I20260926 11:35:39.932454 5214 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:39.932472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34677
--builtin_ntp_servers=127.0.98.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 11:35:39.932947 5214 heartbeater.cc:511] Master 127.0.98.62:34677 requested a full tablet report, sending...
I20260926 11:35:39.934764 5025 ts_manager.cc:194] Registered new tserver with Master: 7414624177a74d708aea0127584ad9f4 (127.0.98.1:34779)
W20260926 11:35:40.054056 5217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:40.054399 5217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:40.054481 5217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:40.059051 5217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:40.059185 5217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:40.059329 5217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:40.065560 5217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:40.066896 5217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:40.068521 5217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:40.078704 5223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:40.078766 5225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:40.078728 5222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:40.079044 5217 server_base.cc:1034] running on GCE node
I20260926 11:35:40.079669 5217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:40.080365 5217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:40.081585 5217 hybrid_clock.cc:648] HybridClock initialized: now 1790422540081543 us; error 60 us; skew 500 ppm
I20260926 11:35:40.083772 5217 webserver.cc:460] Webserver started at http://127.0.98.2:34731/ using document root <none> and password file <none>
I20260926 11:35:40.084424 5217 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:40.084527 5217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:40.084759 5217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:40.086920 5217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973"
format_stamp: "Formatted at 2026-09-26 11:35:40 on dist-test-slave-4pw5"
server_key: "bf9d51cb6dc60dc09c6fcdb5bc04c861"
server_key_iv: "9e65b1b63f6567efb96ffb073fe49803"
server_key_version: "encryptionkey@0"
I20260926 11:35:40.087540 5217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973"
format_stamp: "Formatted at 2026-09-26 11:35:40 on dist-test-slave-4pw5"
server_key: "bf9d51cb6dc60dc09c6fcdb5bc04c861"
server_key_iv: "9e65b1b63f6567efb96ffb073fe49803"
server_key_version: "encryptionkey@0"
I20260926 11:35:40.091555 5217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:35:40.094329 5231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:40.095778 5217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 11:35:40.095921 5217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973"
format_stamp: "Formatted at 2026-09-26 11:35:40 on dist-test-slave-4pw5"
server_key: "bf9d51cb6dc60dc09c6fcdb5bc04c861"
server_key_iv: "9e65b1b63f6567efb96ffb073fe49803"
server_key_version: "encryptionkey@0"
I20260926 11:35:40.096056 5217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:40.101081 5217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:40.101737 5217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:40.101938 5217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:40.102557 5217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:40.103533 5217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:40.103647 5217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:40.103741 5217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:40.103796 5217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:40.115662 5217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34879
I20260926 11:35:40.115685 5344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34879 every 8 connection(s)
I20260926 11:35:40.116724 5217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:40.119974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 5217
I20260926 11:35:40.120119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:40.120458 392 external_mini_cluster.cc:1428] Setting key 95b77be147ec27eab645e79f962ee24b
I20260926 11:35:40.123119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34677
--builtin_ntp_servers=127.0.98.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 11:35:40.124146 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.118790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57287 (local address 127.0.98.62:34677)
0926 11:35:40.118965 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.118970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.120364 (+ 1394us) server_negotiation.cc:407] Connection header received
0926 11:35:40.121962 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.121965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.122001 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.122117 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.123618 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.123622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.123625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.123645 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.123666 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.123676 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.123757 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.123973 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.124018 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:35:40.125180 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.119223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34677 (local address 127.0.98.2:57287)
0926 11:35:40.120171 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:40.120232 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:40.121711 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:40.122331 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:40.122340 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:40.122458 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:40.123233 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:35:40.123237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:40.123303 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:35:40.123398 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:40.123770 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:40.123782 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:40.123791 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 11:35:40.123982 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 11:35:40.124268 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":126,"threads_started":1}
I20260926 11:35:40.126554 5345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34677
I20260926 11:35:40.126816 5345 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:40.127411 5345 heartbeater.cc:511] Master 127.0.98.62:34677 requested a full tablet report, sending...
I20260926 11:35:40.128821 5025 ts_manager.cc:194] Registered new tserver with Master: bfe3d0ccb58e4b03b93ebfafb97fe973 (127.0.98.2:34879)
W20260926 11:35:40.244432 5348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:40.244745 5348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:40.244788 5348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:40.249471 5348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:40.249709 5348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:40.249879 5348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:40.255877 5348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:40.257114 5348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:40.258605 5348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:40.269784 5354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:40.269795 5356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:40.269784 5353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:40.270303 5348 server_base.cc:1034] running on GCE node
I20260926 11:35:40.270821 5348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:40.271531 5348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:40.272758 5348 hybrid_clock.cc:648] HybridClock initialized: now 1790422540272741 us; error 32 us; skew 500 ppm
I20260926 11:35:40.274844 5348 webserver.cc:460] Webserver started at http://127.0.98.3:42043/ using document root <none> and password file <none>
I20260926 11:35:40.275537 5348 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:40.275696 5348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:40.275941 5348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:40.278226 5348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "4520971fcf074a668e079f37f62176b2"
format_stamp: "Formatted at 2026-09-26 11:35:40 on dist-test-slave-4pw5"
server_key: "a34ac361b6136994c2e1d30333bb7a0f"
server_key_iv: "16f9743a9bd0457c78b34f142d771c66"
server_key_version: "encryptionkey@0"
I20260926 11:35:40.278869 5348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4520971fcf074a668e079f37f62176b2"
format_stamp: "Formatted at 2026-09-26 11:35:40 on dist-test-slave-4pw5"
server_key: "a34ac361b6136994c2e1d30333bb7a0f"
server_key_iv: "16f9743a9bd0457c78b34f142d771c66"
server_key_version: "encryptionkey@0"
I20260926 11:35:40.283021 5348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 11:35:40.285972 5362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:40.287541 5348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:40.287755 5348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "4520971fcf074a668e079f37f62176b2"
format_stamp: "Formatted at 2026-09-26 11:35:40 on dist-test-slave-4pw5"
server_key: "a34ac361b6136994c2e1d30333bb7a0f"
server_key_iv: "16f9743a9bd0457c78b34f142d771c66"
server_key_version: "encryptionkey@0"
I20260926 11:35:40.287894 5348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:40.293376 5348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:40.294065 5348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:40.294275 5348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:40.294849 5348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:40.295874 5348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:40.295945 5348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:40.296016 5348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:40.296066 5348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:40.305635 5348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46347
I20260926 11:35:40.305656 5475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46347 every 8 connection(s)
I20260926 11:35:40.306661 5348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:40.311081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 5348
I20260926 11:35:40.311223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:40.311544 392 external_mini_cluster.cc:1428] Setting key 8960e94b9c3943bee8cbf92919915025
I20260926 11:35:40.314731 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.308973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32771 (local address 127.0.98.62:34677)
0926 11:35:40.309124 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.309128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.310414 (+ 1286us) server_negotiation.cc:407] Connection header received
0926 11:35:40.312367 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.312372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.312450 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.312558 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.314028 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.314031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.314034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.314048 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.314076 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.314086 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.314194 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.314566 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.314610 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:35:40.315536 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.309447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34677 (local address 127.0.98.3:32771)
0926 11:35:40.310244 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:40.310289 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:40.312077 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:40.312819 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:40.312829 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:40.312963 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:40.313697 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:35:40.313699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:40.313744 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:35:40.313819 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:40.314200 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:40.314208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:40.314214 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:35:40.314426 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 11:35:40.314683 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":166,"threads_started":1}
I20260926 11:35:40.316740 5476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34677
I20260926 11:35:40.316993 5476 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:40.317433 5476 heartbeater.cc:511] Master 127.0.98.62:34677 requested a full tablet report, sending...
I20260926 11:35:40.318506 5025 ts_manager.cc:194] Registered new tserver with Master: 4520971fcf074a668e079f37f62176b2 (127.0.98.3:46347)
I20260926 11:35:40.326045 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:35:40.329632 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.328056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.0.98.62:34677)
0926 11:35:40.328213 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.328218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.328324 (+ 106us) server_negotiation.cc:407] Connection header received
0926 11:35:40.328421 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.328423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.328507 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.328583 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.329230 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.329232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.329234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.329245 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.329262 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.329269 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.329339 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.329376 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.329410 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:35:40.333185 5025 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:40.336577 5025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:40.349493 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.346129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.0.98.2:34879)
0926 11:35:40.347437 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.347452 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.347494 (+ 42us) server_negotiation.cc:407] Connection header received
0926 11:35:40.347571 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.347576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.347787 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.347947 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.348924 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.348936 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.348946 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.348956 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.348995 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.349017 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.349133 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.349246 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.349321 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":89,"threads_started":1}
I20260926 11:35:40.349478 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.345971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34879 (local address 127.0.0.1:54570)
0926 11:35:40.346890 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:40.346947 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:40.347069 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:40.347939 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:40.347945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:40.348033 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:40.348715 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:35:40.348717 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:40.348733 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:35:40.348788 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:40.349113 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:40.349121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:40.349127 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:35:40.349234 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 11:35:40.349310 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"spinlock_wait_cycles":8192,"thread_start_us":140,"threads_started":1}
I20260926 11:35:40.350543 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.346825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34779 (local address 127.0.0.1:42874)
0926 11:35:40.347265 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:40.347277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:40.347343 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:40.348696 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:40.348700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:40.348718 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:40.349408 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:35:40.349411 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:40.349423 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:35:40.349453 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:40.350257 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:40.350263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:40.350266 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 11:35:40.350349 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 11:35:40.350399 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":94,"threads_started":1}
I20260926 11:35:40.350733 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.346280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.0.98.3:46347)
0926 11:35:40.346870 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.346888 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.347558 (+ 670us) server_negotiation.cc:407] Connection header received
0926 11:35:40.347650 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.347658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.347858 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.348003 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.350028 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.350034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.350045 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.350057 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.350096 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.350135 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.350234 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.350432 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.350517 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":182,"threads_started":1}
I20260926 11:35:40.351032 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.345971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46347 (local address 127.0.0.1:36894)
0926 11:35:40.346890 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:40.346947 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:40.347069 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:40.348701 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:40.348704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:40.348718 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:40.349845 (+ 1127us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:35:40.349847 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:40.349858 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:35:40.349883 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:40.350271 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:40.350274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:40.350277 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 11:35:40.350349 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 11:35:40.350896 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":609,"spinlock_wait_cycles":3072,"thread_start_us":117,"threads_started":1}
I20260926 11:35:40.351145 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.347155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.0.98.1:34779)
0926 11:35:40.347797 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.347808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.347830 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:35:40.347882 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.347887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.348053 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.348178 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.349958 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.349967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.349982 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.349995 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.350054 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.350092 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.350210 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.350795 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.350938 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":114,"threads_started":1}
I20260926 11:35:40.352849 5279 tablet_service.cc:1467] Processing CreateTablet for tablet c6e32cb06857425e84f839d4b5028b95 (DEFAULT_TABLE table=test-table [id=d92f9fc9079f4ac6aa8034d238c61492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:40.354091 5279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e32cb06857425e84f839d4b5028b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:40.354029 5410 tablet_service.cc:1467] Processing CreateTablet for tablet c6e32cb06857425e84f839d4b5028b95 (DEFAULT_TABLE table=test-table [id=d92f9fc9079f4ac6aa8034d238c61492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:40.354074 5148 tablet_service.cc:1467] Processing CreateTablet for tablet c6e32cb06857425e84f839d4b5028b95 (DEFAULT_TABLE table=test-table [id=d92f9fc9079f4ac6aa8034d238c61492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:40.355151 5148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e32cb06857425e84f839d4b5028b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:40.355466 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e32cb06857425e84f839d4b5028b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:40.361804 5493 tablet_bootstrap.cc:492] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: Bootstrap starting.
I20260926 11:35:40.363251 5494 tablet_bootstrap.cc:492] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: Bootstrap starting.
I20260926 11:35:40.363919 5495 tablet_bootstrap.cc:492] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4: Bootstrap starting.
I20260926 11:35:40.365156 5493 tablet_bootstrap.cc:654] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:40.365505 5494 tablet_bootstrap.cc:654] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:40.365842 5495 tablet_bootstrap.cc:654] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:40.366163 5493 log.cc:824] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:40.366323 5494 log.cc:824] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:40.366691 5495 log.cc:824] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:40.368582 5493 tablet_bootstrap.cc:492] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: No bootstrap required, opened a new log
I20260926 11:35:40.368664 5494 tablet_bootstrap.cc:492] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: No bootstrap required, opened a new log
I20260926 11:35:40.368794 5493 ts_tablet_manager.cc:1397] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 11:35:40.368839 5494 ts_tablet_manager.cc:1397] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:35:40.369066 5495 tablet_bootstrap.cc:492] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4: No bootstrap required, opened a new log
I20260926 11:35:40.369233 5495 ts_tablet_manager.cc:1397] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:35:40.372471 5494 raft_consensus.cc:348] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } }
I20260926 11:35:40.372756 5494 raft_consensus.cc:374] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:40.372809 5494 raft_consensus.cc:729] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe3d0ccb58e4b03b93ebfafb97fe973, State: Initialized, Role: FOLLOWER
I20260926 11:35:40.373283 5494 consensus_queue.cc:260] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } }
I20260926 11:35:40.373529 5493 raft_consensus.cc:348] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } }
I20260926 11:35:40.373811 5493 raft_consensus.cc:374] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:40.373868 5493 raft_consensus.cc:729] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4520971fcf074a668e079f37f62176b2, State: Initialized, Role: FOLLOWER
I20260926 11:35:40.373718 5495 raft_consensus.cc:348] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } }
I20260926 11:35:40.373939 5495 raft_consensus.cc:374] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:40.374001 5495 raft_consensus.cc:729] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7414624177a74d708aea0127584ad9f4, State: Initialized, Role: FOLLOWER
I20260926 11:35:40.374158 5345 heartbeater.cc:503] Master 127.0.98.62:34677 was elected leader, sending a full tablet report...
I20260926 11:35:40.374428 5495 consensus_queue.cc:260] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } }
I20260926 11:35:40.374540 5494 ts_tablet_manager.cc:1428] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:35:40.375020 5493 consensus_queue.cc:260] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } }
I20260926 11:35:40.375469 5495 ts_tablet_manager.cc:1428] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:35:40.375898 5476 heartbeater.cc:503] Master 127.0.98.62:34677 was elected leader, sending a full tablet report...
I20260926 11:35:40.376127 5493 ts_tablet_manager.cc:1428] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260926 11:35:40.377017 5214 heartbeater.cc:503] Master 127.0.98.62:34677 was elected leader, sending a full tablet report...
W20260926 11:35:40.423949 5215 tablet.cc:2369] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:40.557744 5477 tablet.cc:2369] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:35:40.567633 5501 raft_consensus.cc:482] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:40.567935 5501 raft_consensus.cc:504] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } }
I20260926 11:35:40.569228 5501 leader_election.cc:288] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfe3d0ccb58e4b03b93ebfafb97fe973 (127.0.98.2:34879), 7414624177a74d708aea0127584ad9f4 (127.0.98.1:34779)
I20260926 11:35:40.571394 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.569647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34879 (local address 127.0.98.3:40027)
0926 11:35:40.569790 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:40.569805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:40.569904 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:40.570194 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:40.570201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:40.570219 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:40.570839 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:35:40.570840 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:40.570850 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:35:40.570874 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:40.571140 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:40.571143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:40.571147 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 11:35:40.571219 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 11:35:40.571259 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 11:35:40.571462 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.569753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40027 (local address 127.0.98.2:34879)
0926 11:35:40.569958 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.569961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.569978 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:35:40.570020 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.570022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.570052 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.570169 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.571012 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.571016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.571017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.571029 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.571045 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.571053 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.571148 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.571235 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.571275 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 11:35:40.571920 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.569934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34779 (local address 127.0.98.3:40747)
0926 11:35:40.570274 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:40.570288 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:40.570380 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:40.570737 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:40.570741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:40.570761 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:40.571230 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:35:40.571231 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:40.571240 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:35:40.571264 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:40.571624 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:40.571630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:40.571636 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:35:40.571741 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 11:35:40.571775 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":137,"threads_started":1}
I20260926 11:35:40.571952 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.570410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40747 (local address 127.0.98.1:34779)
0926 11:35:40.570559 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.570563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.570576 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:35:40.570612 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.570614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.570643 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.570704 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.571403 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.571406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.571409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.571422 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.571446 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.571457 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.571556 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.571772 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.571826 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:35:40.572163 5299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e32cb06857425e84f839d4b5028b95" candidate_uuid: "4520971fcf074a668e079f37f62176b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" is_pre_election: true
I20260926 11:35:40.572758 5299 raft_consensus.cc:2454] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4520971fcf074a668e079f37f62176b2 in term 0.
I20260926 11:35:40.572990 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e32cb06857425e84f839d4b5028b95" candidate_uuid: "4520971fcf074a668e079f37f62176b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7414624177a74d708aea0127584ad9f4" is_pre_election: true
I20260926 11:35:40.573305 5168 raft_consensus.cc:2454] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4520971fcf074a668e079f37f62176b2 in term 0.
I20260926 11:35:40.573405 5364 leader_election.cc:302] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4520971fcf074a668e079f37f62176b2, bfe3d0ccb58e4b03b93ebfafb97fe973; no voters:
I20260926 11:35:40.573689 5501 raft_consensus.cc:2781] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:40.573782 5501 raft_consensus.cc:482] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:40.573827 5501 raft_consensus.cc:3037] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:40.574952 5501 raft_consensus.cc:504] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } }
I20260926 11:35:40.575352 5501 leader_election.cc:288] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [CANDIDATE]: Term 1 election: Requested vote from peers bfe3d0ccb58e4b03b93ebfafb97fe973 (127.0.98.2:34879), 7414624177a74d708aea0127584ad9f4 (127.0.98.1:34779)
I20260926 11:35:40.575851 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e32cb06857425e84f839d4b5028b95" candidate_uuid: "4520971fcf074a668e079f37f62176b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7414624177a74d708aea0127584ad9f4"
I20260926 11:35:40.575851 5299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e32cb06857425e84f839d4b5028b95" candidate_uuid: "4520971fcf074a668e079f37f62176b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973"
I20260926 11:35:40.575990 5168 raft_consensus.cc:3037] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:40.576018 5299 raft_consensus.cc:3037] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:40.577356 5168 raft_consensus.cc:2454] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4520971fcf074a668e079f37f62176b2 in term 1.
I20260926 11:35:40.577356 5299 raft_consensus.cc:2454] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4520971fcf074a668e079f37f62176b2 in term 1.
I20260926 11:35:40.577699 5364 leader_election.cc:302] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4520971fcf074a668e079f37f62176b2, 7414624177a74d708aea0127584ad9f4; no voters:
I20260926 11:35:40.577945 5501 raft_consensus.cc:2781] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:40.578218 5501 raft_consensus.cc:686] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4520971fcf074a668e079f37f62176b2, State: Running, Role: LEADER
I20260926 11:35:40.578601 5501 consensus_queue.cc:237] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } }
I20260926 11:35:40.582800 5026 catalog_manager.cc:5495] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4520971fcf074a668e079f37f62176b2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4520971fcf074a668e079f37f62176b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4520971fcf074a668e079f37f62176b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 } health_report { overall_health: UNKNOWN } } }
W20260926 11:35:40.618065 5346 tablet.cc:2369] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:35:40.648355 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.646510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.0.98.3:46347)
0926 11:35:40.646677 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.646683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.646708 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:35:40.646859 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.646862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.646937 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.647035 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.647875 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.647880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.647883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.647896 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.647917 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.647928 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.648026 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.648159 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.648204 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:35:40.655516 5299 raft_consensus.cc:1261] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [term 1 FOLLOWER]: Refusing update from remote peer 4520971fcf074a668e079f37f62176b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:40.655516 5168 raft_consensus.cc:1261] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4 [term 1 FOLLOWER]: Refusing update from remote peer 4520971fcf074a668e079f37f62176b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:40.656431 5503 consensus_queue.cc:1035] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7414624177a74d708aea0127584ad9f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:40.656663 5501 consensus_queue.cc:1035] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe3d0ccb58e4b03b93ebfafb97fe973" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:40.667109 5508 mvcc.cc:205] Tried to move back new op lower bound from 7333570726515478528 to 7333570726212734976. Current Snapshot: MvccSnapshot[applied={T|T < 7333570726515478528 or (T in {7333570726515478528})}]
I20260926 11:35:40.673717 5026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:40.673939 5026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:40.676546 5026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc93a3d909ba412a984b1c3a2db593f4: Sending DeleteTablet for 3 replicas of tablet c6e32cb06857425e84f839d4b5028b95
I20260926 11:35:40.677547 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e32cb06857425e84f839d4b5028b95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:40 UTC) from {username='slave'} at 127.0.0.1:36894
I20260926 11:35:40.677590 5148 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e32cb06857425e84f839d4b5028b95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:40 UTC) from {username='slave'} at 127.0.0.1:42874
I20260926 11:35:40.678154 5516 tablet_replica.cc:330] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: stopping tablet replica
I20260926 11:35:40.678143 5279 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e32cb06857425e84f839d4b5028b95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:40 UTC) from {username='slave'} at 127.0.0.1:54570
I20260926 11:35:40.678469 5517 tablet_replica.cc:330] T c6e32cb06857425e84f839d4b5028b95 P 7414624177a74d708aea0127584ad9f4: stopping tablet replica
I20260926 11:35:40.678632 5518 tablet_replica.cc:330] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: stopping tablet replica
I20260926 11:35:40.678679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 5086
I20260926 11:35:40.680572 5518 raft_consensus.cc:2229] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:40.680850 5516 raft_consensus.cc:2229] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:40.680967 5518 raft_consensus.cc:2258] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:40.681244 5516 raft_consensus.cc:2258] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:40.683145 5518 ts_tablet_manager.cc:1905] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:40.683347 5516 ts_tablet_manager.cc:1905] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:40.686285 5518 ts_tablet_manager.cc:1918] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:40.686483 5518 log.cc:1196] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/c6e32cb06857425e84f839d4b5028b95
I20260926 11:35:40.686877 5516 ts_tablet_manager.cc:1918] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:40.686900 5518 ts_tablet_manager.cc:1939] T c6e32cb06857425e84f839d4b5028b95 P bfe3d0ccb58e4b03b93ebfafb97fe973: Deleting consensus metadata
I20260926 11:35:40.686981 5516 log.cc:1196] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/c6e32cb06857425e84f839d4b5028b95
I20260926 11:35:40.687315 5516 ts_tablet_manager.cc:1939] T c6e32cb06857425e84f839d4b5028b95 P 4520971fcf074a668e079f37f62176b2: Deleting consensus metadata
W20260926 11:35:40.687301 5011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34779 (error 108)
I20260926 11:35:40.688202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 5217
I20260926 11:35:40.689123 5013 catalog_manager.cc:4841] TS 4520971fcf074a668e079f37f62176b2 (127.0.98.3:46347): tablet c6e32cb06857425e84f839d4b5028b95 (table test-table [id=d92f9fc9079f4ac6aa8034d238c61492]) successfully deleted
I20260926 11:35:40.693794 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.693417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34779 (local address 127.0.0.1:42890)
0926 11:35:40.693587 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:40.693679 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 11:35:40.694084 5011 catalog_manager.cc:4584] TS 7414624177a74d708aea0127584ad9f4 (127.0.98.1:34779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6e32cb06857425e84f839d4b5028b95: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34779: connect: Connection refused (error 111)
I20260926 11:35:40.696149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 5348
I20260926 11:35:40.697530 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.696177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34879 (local address 127.0.0.1:54580)
0926 11:35:40.697394 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:40.697446 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1138}
I20260926 11:35:40.702410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 4995
2026-09-26T11:35:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1266 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 11:35:40.726271 392 mini_oidc.cc:152] Starting JWKS server
I20260926 11:35:40.728228 392 webserver.cc:460] Webserver started at https://127.0.0.1:36979/ using document root <none> and password file <none>
I20260926 11:35:40.728454 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 11:35:40.728777 392 webserver.cc:460] Webserver started at http://127.0.0.1:39901/ using document root <none> and password file <none>
2026-09-26T11:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:40Z Disabled control of system clock
I20260926 11:35:40.739730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40711
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40711
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36979/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/testchainca.cert with env {}
W20260926 11:35:40.862762 5525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:40.863109 5525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:40.863155 5525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:40.867808 5525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:40.867887 5525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:40.867902 5525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:40.867920 5525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:40.867935 5525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 11:35:40.867952 5525 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 11:35:40.867993 5525 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36979/jwks.json
I20260926 11:35:40.873906 5525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36979/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:40.875104 5525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:40.876278 5525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:40.885301 5533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:40.885304 5530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:40.885351 5525 server_base.cc:1034] running on GCE node
W20260926 11:35:40.885304 5531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:40.886346 5525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:40.887748 5525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:40.888970 5525 hybrid_clock.cc:648] HybridClock initialized: now 1790422540888956 us; error 46 us; skew 500 ppm
I20260926 11:35:40.891129 5525 webserver.cc:460] Webserver started at http://127.0.98.62:37769/ using document root <none> and password file <none>
I20260926 11:35:40.891911 5525 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:40.892014 5525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:40.892274 5525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:40.894806 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "47760eaeecd441cf982532e8d7b63588"
format_stamp: "Formatted at 2026-09-26 11:35:40 on dist-test-slave-4pw5"
server_key: "61d7402efb0f5a65f4205229213737d7"
server_key_iv: "5a01d58048739022f10ee271aca4cf75"
server_key_version: "encryptionkey@0"
I20260926 11:35:40.895539 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "47760eaeecd441cf982532e8d7b63588"
format_stamp: "Formatted at 2026-09-26 11:35:40 on dist-test-slave-4pw5"
server_key: "61d7402efb0f5a65f4205229213737d7"
server_key_iv: "5a01d58048739022f10ee271aca4cf75"
server_key_version: "encryptionkey@0"
I20260926 11:35:40.900056 5525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:40.904075 5539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:40.905774 5525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:35:40.905913 5525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "47760eaeecd441cf982532e8d7b63588"
format_stamp: "Formatted at 2026-09-26 11:35:40 on dist-test-slave-4pw5"
server_key: "61d7402efb0f5a65f4205229213737d7"
server_key_iv: "5a01d58048739022f10ee271aca4cf75"
server_key_version: "encryptionkey@0"
I20260926 11:35:40.906031 5525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:40.961656 5525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:40.962411 5525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:40.962769 5525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:40.971380 5525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40711
I20260926 11:35:40.971541 5591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40711 every 8 connection(s)
I20260926 11:35:40.972819 5525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:40.976689 5592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:40.978169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 5525
I20260926 11:35:40.978302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:40.978595 392 external_mini_cluster.cc:1428] Setting key 4bfd6a04d125704fde0a78030b1d1dfd
I20260926 11:35:40.983402 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588: Bootstrap starting.
I20260926 11:35:40.985832 5592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:40.986624 5592 log.cc:824] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:40.988814 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:40.980240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.0.98.62:40711)
0926 11:35:40.980990 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:40.981006 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:40.981044 (+ 38us) server_negotiation.cc:407] Connection header received
0926 11:35:40.981699 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:40.981716 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:40.982143 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:40.982510 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:40.982950 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:40.984007 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:40.984942 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:40.985465 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:40.985645 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:40.986703 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:40.986709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:40.986723 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:40.987296 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:40.987378 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:40.987408 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:40.987521 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:40.987717 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 11:35:40.988039 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":165,"threads_started":1}
I20260926 11:35:40.988979 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588: No bootstrap required, opened a new log
I20260926 11:35:40.992082 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:35:40.993412 5592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47760eaeecd441cf982532e8d7b63588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } }
I20260926 11:35:40.993703 5592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:40.993790 5592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47760eaeecd441cf982532e8d7b63588, State: Initialized, Role: FOLLOWER
I20260926 11:35:40.994300 5592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47760eaeecd441cf982532e8d7b63588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } }
I20260926 11:35:40.994573 5592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:40.994685 5592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:40.994817 5592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:40.995952 5592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47760eaeecd441cf982532e8d7b63588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } }
I20260926 11:35:40.996330 5592 leader_election.cc:302] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47760eaeecd441cf982532e8d7b63588; no voters:
I20260926 11:35:40.996680 5592 leader_election.cc:288] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:40.996873 5597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:40.997130 5597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [term 1 LEADER]: Becoming Leader. State: Replica: 47760eaeecd441cf982532e8d7b63588, State: Running, Role: LEADER
I20260926 11:35:40.997486 5597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47760eaeecd441cf982532e8d7b63588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } }
I20260926 11:35:40.998113 5592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:40.999688 5598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47760eaeecd441cf982532e8d7b63588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47760eaeecd441cf982532e8d7b63588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } } }
I20260926 11:35:40.999742 5599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47760eaeecd441cf982532e8d7b63588. Latest consensus state: current_term: 1 leader_uuid: "47760eaeecd441cf982532e8d7b63588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47760eaeecd441cf982532e8d7b63588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } } }
I20260926 11:35:40.999825 5598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:40.999856 5599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:41.000252 5607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:41.003993 5607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:41.010972 5607 catalog_manager.cc:1347] Generated new cluster ID: 05e74e859ed64a18a2ef11f8d4e9e723
I20260926 11:35:41.011066 5607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:41.038611 5607 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:41.040066 5607 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:41.047055 5607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47760eaeecd441cf982532e8d7b63588: Generated new TSK 0
I20260926 11:35:41.047950 5607 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 11:35:41.075529 5622 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 11:35:41.075711 5622 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 11:35:41.076407 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:41.058661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.0.98.62:40711)
0926 11:35:41.058796 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:41.058800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:41.058973 (+ 173us) server_negotiation.cc:407] Connection header received
0926 11:35:41.059234 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:41.059237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:41.059291 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:41.059409 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0926 11:35:41.059746 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.060301 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.061234 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.061488 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.061599 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:41.061996 (+ 397us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 11:35:41.075871 (+ 13875us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 11:35:41.076245 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 11:35:41.076507 5595 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 11:35:41.077600 5621 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:41.058536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40711 (local address 127.0.0.1:46758)
0926 11:35:41.058870 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:41.058887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:41.059052 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:41.059422 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:41.059425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:41.059433 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 11:35:41.059627 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:41.059635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.060429 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.060433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:41.061101 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:41.061109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.061602 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.061627 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:41.061730 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:41.061778 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 11:35:41.076155 (+ 14377us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 11:35:41.076413 (+ 258us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":110,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:40711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 11:35:41.078855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 5525
2026-09-26T11:35:41Z chronyd exiting
I20260926 11:35:41.117692 392 test_util.cc:175] -----------------------------------------------
I20260926 11:35:41.117811 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422511662684-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (393 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 11:35:41.118940 392 mini_oidc.cc:152] Starting JWKS server
I20260926 11:35:41.119918 392 webserver.cc:460] Webserver started at https://127.0.0.1:32811/ using document root <none> and password file <none>
I20260926 11:35:41.120048 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 11:35:41.120312 392 webserver.cc:460] Webserver started at http://127.0.0.1:34623/ using document root <none> and password file <none>
2026-09-26T11:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:41Z Disabled control of system clock
I20260926 11:35:41.130194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32863
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32863
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32811/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/testchainca.cert with env {}
W20260926 11:35:41.259874 5628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:41.260198 5628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:41.260241 5628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:41.264565 5628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:41.264649 5628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:41.264664 5628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:41.264681 5628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:41.264698 5628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 11:35:41.264765 5628 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 11:35:41.264793 5628 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32811/jwks.json
I20260926 11:35:41.270638 5628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32811/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:41.271886 5628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:41.273047 5628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:41.282274 5633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:41.282294 5636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:41.282320 5634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:41.282557 5628 server_base.cc:1034] running on GCE node
I20260926 11:35:41.283473 5628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:41.284873 5628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:41.286082 5628 hybrid_clock.cc:648] HybridClock initialized: now 1790422541286054 us; error 49 us; skew 500 ppm
I20260926 11:35:41.288575 5628 webserver.cc:460] Webserver started at http://127.0.98.62:40107/ using document root <none> and password file <none>
I20260926 11:35:41.289215 5628 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:41.289314 5628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:41.289566 5628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:41.292299 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "6999bedde0cb42dfbf80eb27b983fca9"
format_stamp: "Formatted at 2026-09-26 11:35:41 on dist-test-slave-4pw5"
server_key: "7d732e3737242b216890d712ac7cfbd8"
server_key_iv: "e2eb9dd8d79ee5031a82cf529cddedce"
server_key_version: "encryptionkey@0"
I20260926 11:35:41.292996 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "6999bedde0cb42dfbf80eb27b983fca9"
format_stamp: "Formatted at 2026-09-26 11:35:41 on dist-test-slave-4pw5"
server_key: "7d732e3737242b216890d712ac7cfbd8"
server_key_iv: "e2eb9dd8d79ee5031a82cf529cddedce"
server_key_version: "encryptionkey@0"
I20260926 11:35:41.297602 5628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:35:41.301008 5642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:41.302627 5628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:35:41.302829 5628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "6999bedde0cb42dfbf80eb27b983fca9"
format_stamp: "Formatted at 2026-09-26 11:35:41 on dist-test-slave-4pw5"
server_key: "7d732e3737242b216890d712ac7cfbd8"
server_key_iv: "e2eb9dd8d79ee5031a82cf529cddedce"
server_key_version: "encryptionkey@0"
I20260926 11:35:41.302981 5628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:41.355005 5628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:41.356153 5628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:41.356426 5628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:41.365273 5628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32863
I20260926 11:35:41.365279 5694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32863 every 8 connection(s)
I20260926 11:35:41.366504 5628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:41.368669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 5628
I20260926 11:35:41.368818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:41.369139 392 external_mini_cluster.cc:1428] Setting key 5759041d1d0e010b42bafd388656d1f2
I20260926 11:35:41.370769 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:41.377573 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9: Bootstrap starting.
I20260926 11:35:41.380637 5695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:41.380836 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:41.370832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.0.98.62:32863)
0926 11:35:41.371744 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:41.371765 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:41.371816 (+ 51us) server_negotiation.cc:407] Connection header received
0926 11:35:41.372626 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:41.372647 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:41.373192 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:41.373621 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:41.374174 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.375338 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.376476 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.377076 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.377298 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:41.378529 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:41.378535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:41.378552 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:41.379061 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:41.379131 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:41.379156 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:41.379276 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:41.379417 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 11:35:41.379782 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":181,"threads_started":1}
I20260926 11:35:41.381778 5695 log.cc:824] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:41.384493 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9: No bootstrap required, opened a new log
I20260926 11:35:41.384588 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:35:41.389091 5695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6999bedde0cb42dfbf80eb27b983fca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } }
I20260926 11:35:41.389490 5695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:41.389590 5695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6999bedde0cb42dfbf80eb27b983fca9, State: Initialized, Role: FOLLOWER
I20260926 11:35:41.390213 5695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6999bedde0cb42dfbf80eb27b983fca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } }
I20260926 11:35:41.390398 5695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:41.390497 5695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:41.390640 5695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:41.392006 5695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6999bedde0cb42dfbf80eb27b983fca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } }
I20260926 11:35:41.392550 5695 leader_election.cc:302] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6999bedde0cb42dfbf80eb27b983fca9; no voters:
I20260926 11:35:41.393095 5695 leader_election.cc:288] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:41.393253 5700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:41.393502 5700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [term 1 LEADER]: Becoming Leader. State: Replica: 6999bedde0cb42dfbf80eb27b983fca9, State: Running, Role: LEADER
I20260926 11:35:41.393915 5700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6999bedde0cb42dfbf80eb27b983fca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } }
I20260926 11:35:41.394807 5695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:41.396454 5702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6999bedde0cb42dfbf80eb27b983fca9. Latest consensus state: current_term: 1 leader_uuid: "6999bedde0cb42dfbf80eb27b983fca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6999bedde0cb42dfbf80eb27b983fca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } } }
I20260926 11:35:41.396512 5701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6999bedde0cb42dfbf80eb27b983fca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6999bedde0cb42dfbf80eb27b983fca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } } }
I20260926 11:35:41.396600 5702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:41.396626 5701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:41.397243 5710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:41.401342 5710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:41.409861 5710 catalog_manager.cc:1347] Generated new cluster ID: bc24e3da1b274113a186ecf97ff5385c
I20260926 11:35:41.409982 5710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:41.420464 5710 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:41.421834 5710 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:41.429220 5710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6999bedde0cb42dfbf80eb27b983fca9: Generated new TSK 0
I20260926 11:35:41.430191 5710 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 11:35:41.463294 5725 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 11:35:41.463441 5725 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 11:35:41.464399 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:41.446443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.0.98.62:32863)
0926 11:35:41.446631 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:41.446638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:41.446868 (+ 230us) server_negotiation.cc:407] Connection header received
0926 11:35:41.447129 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:41.447133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:41.447202 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:41.447382 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0926 11:35:41.447711 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.448366 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.449827 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.450135 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.450251 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:41.450679 (+ 428us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 11:35:41.463896 (+ 13217us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 11:35:41.464234 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53324: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 11:35:41.464509 5698 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53324: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 11:35:41.465066 5724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:41.446354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32863 (local address 127.0.0.1:53324)
0926 11:35:41.446750 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:41.446767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:41.446917 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:41.447392 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:41.447396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:41.447405 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 11:35:41.447548 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:41.447555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.448503 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.448507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:41.449599 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:41.449615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.450258 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.450293 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:41.450399 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:41.450437 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 11:35:41.464202 (+ 13765us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 11:35:41.464458 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:32863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":159,"threads_started":1}
I20260926 11:35:41.465804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 5628
2026-09-26T11:35:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (387 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 11:35:41.506875 392 mini_oidc.cc:152] Starting JWKS server
I20260926 11:35:41.507942 392 webserver.cc:460] Webserver started at https://127.0.0.1:45437/ using document root <none> and password file <none>
I20260926 11:35:41.508033 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 11:35:41.508257 392 webserver.cc:460] Webserver started at http://127.0.0.1:34623/ using document root <none> and password file <none>
2026-09-26T11:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:41Z Disabled control of system clock
I20260926 11:35:41.519028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45203
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45203
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45437/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 11:35:41.660640 5731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:41.661065 5731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:41.661116 5731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:41.666182 5731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:41.666289 5731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:41.666308 5731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:41.666330 5731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:41.666348 5731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 11:35:41.666369 5731 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 11:35:41.666384 5731 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45437/jwks.json
I20260926 11:35:41.672219 5731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45437/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:41.673705 5731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:41.675489 5731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:41.685884 5736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:41.686039 5731 server_base.cc:1034] running on GCE node
W20260926 11:35:41.685873 5739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:41.686308 5737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:41.686996 5731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:41.688498 5731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:41.689724 5731 hybrid_clock.cc:648] HybridClock initialized: now 1790422541689703 us; error 68 us; skew 500 ppm
I20260926 11:35:41.692476 5731 webserver.cc:460] Webserver started at http://127.0.98.62:36383/ using document root <none> and password file <none>
I20260926 11:35:41.693253 5731 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:41.693359 5731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:41.693636 5731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:41.696529 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "3c6dc123041b4ba8a830deed8ed8a590"
format_stamp: "Formatted at 2026-09-26 11:35:41 on dist-test-slave-4pw5"
server_key: "8469311af3ff7e1487b5793708a4b554"
server_key_iv: "f9c1443d46bbfd154231323994177805"
server_key_version: "encryptionkey@0"
I20260926 11:35:41.697299 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "3c6dc123041b4ba8a830deed8ed8a590"
format_stamp: "Formatted at 2026-09-26 11:35:41 on dist-test-slave-4pw5"
server_key: "8469311af3ff7e1487b5793708a4b554"
server_key_iv: "f9c1443d46bbfd154231323994177805"
server_key_version: "encryptionkey@0"
I20260926 11:35:41.702690 5731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:35:41.706475 5745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:41.708501 5731 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 11:35:41.708735 5731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "3c6dc123041b4ba8a830deed8ed8a590"
format_stamp: "Formatted at 2026-09-26 11:35:41 on dist-test-slave-4pw5"
server_key: "8469311af3ff7e1487b5793708a4b554"
server_key_iv: "f9c1443d46bbfd154231323994177805"
server_key_version: "encryptionkey@0"
I20260926 11:35:41.709009 5731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:41.724416 5731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:41.725477 5731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:41.725752 5731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:41.735396 5731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45203
I20260926 11:35:41.735399 5797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45203 every 8 connection(s)
I20260926 11:35:41.736971 5731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:41.741607 5798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:41.747303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 5731
I20260926 11:35:41.747434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:41.747825 392 external_mini_cluster.cc:1428] Setting key ae431b30d9d5543ead9f531d228e9f7e
I20260926 11:35:41.751344 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590: Bootstrap starting.
I20260926 11:35:41.755086 5798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:41.756592 5798 log.cc:824] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:41.759965 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590: No bootstrap required, opened a new log
I20260926 11:35:41.761240 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:41.750131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.0.98.62:45203)
0926 11:35:41.751128 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:41.751150 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:41.751201 (+ 51us) server_negotiation.cc:407] Connection header received
0926 11:35:41.752168 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:41.752196 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:41.752778 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:41.753190 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:41.753694 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.754834 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.756486 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.757363 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.757593 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:41.758686 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:41.758693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:41.758714 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:41.759279 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:41.759362 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:41.759389 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:41.759487 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:41.759745 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 11:35:41.760179 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":210,"threads_started":1}
I20260926 11:35:41.765363 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:35:41.765136 5798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6dc123041b4ba8a830deed8ed8a590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45203 } }
I20260926 11:35:41.765503 5798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:41.765596 5798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6dc123041b4ba8a830deed8ed8a590, State: Initialized, Role: FOLLOWER
I20260926 11:35:41.766176 5798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6dc123041b4ba8a830deed8ed8a590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45203 } }
I20260926 11:35:41.766362 5798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:41.766456 5798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:41.766583 5798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:41.768091 5798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6dc123041b4ba8a830deed8ed8a590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45203 } }
I20260926 11:35:41.768522 5798 leader_election.cc:302] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c6dc123041b4ba8a830deed8ed8a590; no voters:
I20260926 11:35:41.768925 5798 leader_election.cc:288] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:41.769131 5803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:41.769445 5803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [term 1 LEADER]: Becoming Leader. State: Replica: 3c6dc123041b4ba8a830deed8ed8a590, State: Running, Role: LEADER
I20260926 11:35:41.769887 5803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6dc123041b4ba8a830deed8ed8a590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45203 } }
I20260926 11:35:41.770606 5798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:41.772207 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c6dc123041b4ba8a830deed8ed8a590. Latest consensus state: current_term: 1 leader_uuid: "3c6dc123041b4ba8a830deed8ed8a590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6dc123041b4ba8a830deed8ed8a590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45203 } } }
I20260926 11:35:41.772269 5805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c6dc123041b4ba8a830deed8ed8a590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6dc123041b4ba8a830deed8ed8a590" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45203 } } }
I20260926 11:35:41.772409 5805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:41.772330 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:41.772910 5810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:41.777379 5810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:41.785784 5810 catalog_manager.cc:1347] Generated new cluster ID: 7ec0b7cbdd04440e8db3c45d81abac8c
I20260926 11:35:41.785908 5810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:41.802280 5810 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:41.803915 5810 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:41.814500 5810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c6dc123041b4ba8a830deed8ed8a590: Generated new TSK 0
I20260926 11:35:41.815788 5810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:35:41.834052 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:41.827158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.0.98.62:45203)
0926 11:35:41.827352 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:41.827358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:41.827607 (+ 249us) server_negotiation.cc:407] Connection header received
0926 11:35:41.827862 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:41.827865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:41.827939 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:41.828054 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0926 11:35:41.828411 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.829084 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.830604 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.830874 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.830973 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:41.831450 (+ 477us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 11:35:41.833518 (+ 2068us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 11:35:41.833845 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 11:35:41.834163 5801 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 11:35:41.836304 5827 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:41.827049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45203 (local address 127.0.0.1:42758)
0926 11:35:41.827449 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:41.827468 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:41.827660 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:41.828080 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:41.828083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:41.828091 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 11:35:41.828252 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:41.828260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.829263 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.829267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:41.830435 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:41.830449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:41.830984 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:41.831019 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:41.831117 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:41.831161 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 11:35:41.833746 (+ 2585us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 11:35:41.836078 (+ 2332us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":1971,"thread_start_us":157,"threads_started":1}
I20260926 11:35:41.837116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 5731
E20260926 11:35:41.845644 5828 webserver.cc:533] Webserver: sslize failed:
E20260926 11:35:41.845810 5828 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T11:35:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (372 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T11:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:41Z Disabled control of system clock
I20260926 11:35:41.891173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32891
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/testchainca.cert with env {}
W20260926 11:35:42.034668 5832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:42.035065 5832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:42.035110 5832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:42.039630 5832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:42.039716 5832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:42.039732 5832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:42.039752 5832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:42.039767 5832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 11:35:42.039785 5832 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 11:35:42.039799 5832 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 11:35:42.046136 5832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:42.047693 5832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:42.049085 5832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:42.061235 5838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:42.061542 5837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:42.061586 5840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:42.061235 5832 server_base.cc:1034] running on GCE node
I20260926 11:35:42.062686 5832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:42.064677 5832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:42.066080 5832 hybrid_clock.cc:648] HybridClock initialized: now 1790422542066038 us; error 78 us; skew 500 ppm
I20260926 11:35:42.069267 5832 webserver.cc:460] Webserver started at http://127.0.98.62:43367/ using document root <none> and password file <none>
I20260926 11:35:42.070094 5832 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:42.070216 5832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:42.070528 5832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:42.073752 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "700a9b17223c4d9ab1ee86ca112ec456"
format_stamp: "Formatted at 2026-09-26 11:35:42 on dist-test-slave-4pw5"
server_key: "c2cb32ff732d5c36ccf7f35a6dcacbd1"
server_key_iv: "8fae13dcd9d19184d565853ea63431a3"
server_key_version: "encryptionkey@0"
I20260926 11:35:42.074479 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "700a9b17223c4d9ab1ee86ca112ec456"
format_stamp: "Formatted at 2026-09-26 11:35:42 on dist-test-slave-4pw5"
server_key: "c2cb32ff732d5c36ccf7f35a6dcacbd1"
server_key_iv: "8fae13dcd9d19184d565853ea63431a3"
server_key_version: "encryptionkey@0"
I20260926 11:35:42.080092 5832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 11:35:42.084024 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:42.086110 5832 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:42.086367 5832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "700a9b17223c4d9ab1ee86ca112ec456"
format_stamp: "Formatted at 2026-09-26 11:35:42 on dist-test-slave-4pw5"
server_key: "c2cb32ff732d5c36ccf7f35a6dcacbd1"
server_key_iv: "8fae13dcd9d19184d565853ea63431a3"
server_key_version: "encryptionkey@0"
I20260926 11:35:42.086561 5832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:42.105393 5832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:42.106446 5832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:42.106724 5832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:42.116719 5832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32891
I20260926 11:35:42.116719 5898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32891 every 8 connection(s)
I20260926 11:35:42.118285 5832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:42.120435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 5832
I20260926 11:35:42.120597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:42.120975 392 external_mini_cluster.cc:1428] Setting key e8e118d55907761ce6ddd97047e0e1fb
I20260926 11:35:42.123539 5899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:42.133994 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456: Bootstrap starting.
I20260926 11:35:42.136009 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:42.122752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.0.98.62:32891)
0926 11:35:42.123778 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:42.123806 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:42.123867 (+ 61us) server_negotiation.cc:407] Connection header received
0926 11:35:42.125016 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:42.125051 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:42.125813 (+ 762us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:42.126365 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:42.126991 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:42.128919 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:42.130118 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:42.131021 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:42.131228 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:42.132612 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:42.132621 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:42.132646 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:35:42.133431 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:42.133550 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:35:42.133591 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:42.133750 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:42.133957 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 11:35:42.134451 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":178,"threads_started":1}
I20260926 11:35:42.138161 5899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:42.139369 5899 log.cc:824] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:42.140113 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:35:42.142248 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456: No bootstrap required, opened a new log
I20260926 11:35:42.147027 5899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a9b17223c4d9ab1ee86ca112ec456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260926 11:35:42.147437 5899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:42.147500 5899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700a9b17223c4d9ab1ee86ca112ec456, State: Initialized, Role: FOLLOWER
I20260926 11:35:42.148203 5899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a9b17223c4d9ab1ee86ca112ec456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260926 11:35:42.148434 5899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:42.148537 5899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:42.148691 5899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:42.150136 5899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a9b17223c4d9ab1ee86ca112ec456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260926 11:35:42.150624 5899 leader_election.cc:302] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700a9b17223c4d9ab1ee86ca112ec456; no voters:
I20260926 11:35:42.151049 5899 leader_election.cc:288] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:42.151221 5904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:42.151607 5904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [term 1 LEADER]: Becoming Leader. State: Replica: 700a9b17223c4d9ab1ee86ca112ec456, State: Running, Role: LEADER
I20260926 11:35:42.152062 5904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a9b17223c4d9ab1ee86ca112ec456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260926 11:35:42.152899 5899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:42.154381 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700a9b17223c4d9ab1ee86ca112ec456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a9b17223c4d9ab1ee86ca112ec456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260926 11:35:42.154569 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:42.155366 5906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700a9b17223c4d9ab1ee86ca112ec456. Latest consensus state: current_term: 1 leader_uuid: "700a9b17223c4d9ab1ee86ca112ec456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a9b17223c4d9ab1ee86ca112ec456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260926 11:35:42.156311 5906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456 [sys.catalog]: This master's current role is: LEADER
W20260926 11:35:42.160233 5920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:35:42.160389 5920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:35:42.160507 5913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:42.161618 5913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:42.169373 5913 catalog_manager.cc:1347] Generated new cluster ID: 13d359564bc24ff982e0fa42ab491e66
I20260926 11:35:42.169493 5913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:42.183512 5913 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:42.185251 5913 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:42.192029 5913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 700a9b17223c4d9ab1ee86ca112ec456: Generated new TSK 0
I20260926 11:35:42.193277 5913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:35:42.236670 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:42.208060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.0.98.62:32891)
0926 11:35:42.208246 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:42.208252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:42.208563 (+ 311us) server_negotiation.cc:407] Connection header received
0926 11:35:42.208719 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:42.208724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:42.208801 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:42.208957 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0926 11:35:42.209349 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:42.210297 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:42.211371 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:42.211748 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:42.211860 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:42.212390 (+ 530us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 11:35:42.236038 (+ 23648us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 11:35:42.236471 (+ 433us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59582: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 11:35:42.236799 5902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59582: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 11:35:42.236907 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:42.207900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32891 (local address 127.0.0.1:59582)
0926 11:35:42.208386 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:42.208402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:42.208557 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:42.208988 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:42.208991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:42.209001 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 11:35:42.209142 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:42.209211 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:42.210439 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:42.210443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:42.211241 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:42.211252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:42.211865 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:42.211900 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:42.212003 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:42.212077 (+ 74us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 11:35:42.236401 (+ 24324us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 11:35:42.236667 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:32891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":173,"threads_started":1}
I20260926 11:35:42.237815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 5832
2026-09-26T11:35:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (379 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (153 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (137 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:42 dist-test-slave-4pw5 krb5kdc[5940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:42 dist-test-slave-4pw5 krb5kdc[5940](info): set up 2 sockets
Sep 26 11:35:42 dist-test-slave-4pw5 krb5kdc[5940](info): commencing operation
krb5kdc: starting...
W20260926 11:35:43.604190 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:43 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:43.740124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43081
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:43.856452 5953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:43.856762 5953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:43.856809 5953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:43.861138 5953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:43.861220 5953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:43.861244 5953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:43.861265 5953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:43.861284 5953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:43.867020 5953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:43.868357 5953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:43.869477 5953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:43.879293 5959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:43.879263 5958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:43.879442 5953 server_base.cc:1034] running on GCE node
W20260926 11:35:43.879665 5961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:43.880249 5953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:43.881524 5953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:43.882722 5953 hybrid_clock.cc:648] HybridClock initialized: now 1790422543882692 us; error 44 us; skew 500 ppm
Sep 26 11:35:43 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:43.891726 5953 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:43.892920 5953 webserver.cc:460] Webserver started at http://127.0.98.62:35755/ using document root <none> and password file <none>
I20260926 11:35:43.893601 5953 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:43.893693 5953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:43.893949 5953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:43.896373 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "0b373889f3f3455caaa1efdeb33ea637"
format_stamp: "Formatted at 2026-09-26 11:35:43 on dist-test-slave-4pw5"
server_key: "690d5fca82c101c91f31b38cd5999f8d"
server_key_iv: "09102c7718911e2056dccf623dd33b3e"
server_key_version: "encryptionkey@0"
I20260926 11:35:43.897040 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b373889f3f3455caaa1efdeb33ea637"
format_stamp: "Formatted at 2026-09-26 11:35:43 on dist-test-slave-4pw5"
server_key: "690d5fca82c101c91f31b38cd5999f8d"
server_key_iv: "09102c7718911e2056dccf623dd33b3e"
server_key_version: "encryptionkey@0"
I20260926 11:35:43.901295 5953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:35:43.904248 5968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:43.905733 5953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:43.905932 5953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "0b373889f3f3455caaa1efdeb33ea637"
format_stamp: "Formatted at 2026-09-26 11:35:43 on dist-test-slave-4pw5"
server_key: "690d5fca82c101c91f31b38cd5999f8d"
server_key_iv: "09102c7718911e2056dccf623dd33b3e"
server_key_version: "encryptionkey@0"
I20260926 11:35:43.906086 5953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:43.940162 5953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:43.944484 5953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:43.944746 5953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:43.952428 5953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43081
I20260926 11:35:43.952773 6020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43081 every 8 connection(s)
I20260926 11:35:43.953521 5953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:43.956842 6021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:43.957806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 5953
I20260926 11:35:43.957965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:43.958317 392 external_mini_cluster.cc:1428] Setting key 432775e0a8eb2be3351b99a6ffb3b5a7
I20260926 11:35:43.964293 6021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637: Bootstrap starting.
I20260926 11:35:43.967192 6021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:43.968124 6021 log.cc:824] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:35:43 dist-test-slave-4pw5 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:43.970846 6021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637: No bootstrap required, opened a new log
I20260926 11:35:43.975183 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:43.960131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.0.98.62:43081)
0926 11:35:43.960601 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:43.960621 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:43.960666 (+ 45us) server_negotiation.cc:407] Connection header received
0926 11:35:43.961522 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:43.961545 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:43.962016 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:43.962433 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:43.963214 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:43.964749 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:43.965771 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:43.966477 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:43.966651 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:43.969660 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:43.969667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:43.969684 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:43.969712 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:43.971747 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:43.972309 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:43.972315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:43.972322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:43.972443 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:43.972787 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:43.972791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:43.972792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:43.973288 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:43.973486 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:43.973828 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 11:35:43.974211 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":114,"threads_started":1}
I20260926 11:35:43.975381 6021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b373889f3f3455caaa1efdeb33ea637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } }
I20260926 11:35:43.975759 6021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:43.975853 6021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b373889f3f3455caaa1efdeb33ea637, State: Initialized, Role: FOLLOWER
I20260926 11:35:43.976332 6021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b373889f3f3455caaa1efdeb33ea637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } }
I20260926 11:35:43.976493 6021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:43.976563 6021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:43.976668 6021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:43.977735 6021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b373889f3f3455caaa1efdeb33ea637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } }
I20260926 11:35:43.978299 6021 leader_election.cc:302] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b373889f3f3455caaa1efdeb33ea637; no voters:
I20260926 11:35:43.978674 6021 leader_election.cc:288] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:43.978830 6026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:43.979143 6026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [term 1 LEADER]: Becoming Leader. State: Replica: 0b373889f3f3455caaa1efdeb33ea637, State: Running, Role: LEADER
I20260926 11:35:43.979537 6026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b373889f3f3455caaa1efdeb33ea637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } }
I20260926 11:35:43.980167 6021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:43.981901 6028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b373889f3f3455caaa1efdeb33ea637. Latest consensus state: current_term: 1 leader_uuid: "0b373889f3f3455caaa1efdeb33ea637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b373889f3f3455caaa1efdeb33ea637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } } }
I20260926 11:35:43.982167 6028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:43.982215 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b373889f3f3455caaa1efdeb33ea637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b373889f3f3455caaa1efdeb33ea637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } } }
I20260926 11:35:43.982340 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:43.982815 6041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:43.986171 6041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:43.994158 6041 catalog_manager.cc:1347] Generated new cluster ID: 5092512313c240ac8ab4d67a7ee1c924
I20260926 11:35:43.994266 6041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:44.013064 6041 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:44.014169 6041 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:44.027133 6041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b373889f3f3455caaa1efdeb33ea637: Generated new TSK 0
I20260926 11:35:44.028208 6041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:44.070891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:44.195402 6047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:44.195808 6047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:44.195906 6047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:44.200281 6047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:44.200366 6047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:44.200433 6047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:44.206246 6047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:44.207548 6047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:44.208655 6047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:44.219482 6052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:44.219462 6053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:44.219782 6047 server_base.cc:1034] running on GCE node
W20260926 11:35:44.219952 6055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:44.220475 6047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:44.221182 6047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:44.222368 6047 hybrid_clock.cc:648] HybridClock initialized: now 1790422544222351 us; error 34 us; skew 500 ppm
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.226197 6047 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:44.227372 6047 webserver.cc:460] Webserver started at http://127.0.98.1:42507/ using document root <none> and password file <none>
I20260926 11:35:44.228047 6047 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:44.228117 6047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:44.228344 6047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:44.230577 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "a9b6ed4f0320450cb523df0fb40e62a6"
format_stamp: "Formatted at 2026-09-26 11:35:44 on dist-test-slave-4pw5"
server_key: "09c29f16de298456605c83209e0f4ea7"
server_key_iv: "954cf2750016d54d01c14d8af56b52e2"
server_key_version: "encryptionkey@0"
I20260926 11:35:44.231355 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a9b6ed4f0320450cb523df0fb40e62a6"
format_stamp: "Formatted at 2026-09-26 11:35:44 on dist-test-slave-4pw5"
server_key: "09c29f16de298456605c83209e0f4ea7"
server_key_iv: "954cf2750016d54d01c14d8af56b52e2"
server_key_version: "encryptionkey@0"
I20260926 11:35:44.235741 6047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 11:35:44.238478 6062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:44.239989 6047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:44.240103 6047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "a9b6ed4f0320450cb523df0fb40e62a6"
format_stamp: "Formatted at 2026-09-26 11:35:44 on dist-test-slave-4pw5"
server_key: "09c29f16de298456605c83209e0f4ea7"
server_key_iv: "954cf2750016d54d01c14d8af56b52e2"
server_key_version: "encryptionkey@0"
I20260926 11:35:44.240213 6047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:44.281019 6047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:44.284798 6047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:44.284976 6047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:44.285560 6047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:44.286412 6047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:44.286459 6047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:44.286501 6047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:44.286517 6047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:44.296469 6047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36273
I20260926 11:35:44.296514 6175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36273 every 8 connection(s)
I20260926 11:35:44.297534 6047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:44.299000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 6047
I20260926 11:35:44.299108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:44.299470 392 external_mini_cluster.cc:1428] Setting key 23e8b53cf403ae7c4a76a90ab425648d
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:35:44.315552 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:44.299915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53163 (local address 127.0.98.62:43081)
0926 11:35:44.300084 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:44.300090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:44.301187 (+ 1097us) server_negotiation.cc:407] Connection header received
0926 11:35:44.303324 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:44.303331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:44.303396 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:44.303503 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:44.304879 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.305805 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.307713 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.307954 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.308072 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:44.311271 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:44.311274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:44.311276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:44.311303 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:44.312945 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.313747 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.313753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.313756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.313851 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.314231 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.314235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.314237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.314441 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:44.314562 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:44.315279 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 11:35:44.315418 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:35:44.316730 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:44.300395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43081 (local address 127.0.98.1:53163)
0926 11:35:44.301009 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:44.301074 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:44.303006 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:44.303768 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:44.303776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:44.304167 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:44.304702 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:44.304717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.305951 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.305954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:44.307533 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:44.307551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.308164 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.308170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:44.308401 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:44.309116 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:44.309139 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:44.311055 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:44.313180 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:44.313186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:44.313208 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:44.313626 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:44.313959 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:44.313962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:44.313965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:44.314124 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:44.314561 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:44.314569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:44.314949 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 11:35:44.315199 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 11:35:44.315674 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":133,"threads_started":1}
I20260926 11:35:44.318213 6176 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43081
I20260926 11:35:44.318531 6176 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:44.319248 6176 heartbeater.cc:511] Master 127.0.98.62:43081 requested a full tablet report, sending...
I20260926 11:35:44.321152 5983 ts_manager.cc:194] Registered new tserver with Master: a9b6ed4f0320450cb523df0fb40e62a6 (127.0.98.1:36273)
I20260926 11:35:44.322338 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:35:44.331228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:44.452709 6181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:44.453025 6181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:44.453071 6181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:44.457461 6181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:44.457567 6181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:44.457677 6181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:44.463478 6181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:44.464752 6181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:44.465847 6181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:44.477011 6186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:44.477072 6181 server_base.cc:1034] running on GCE node
W20260926 11:35:44.477015 6189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:44.477015 6187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:44.477694 6181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:44.478444 6181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:44.479673 6181 hybrid_clock.cc:648] HybridClock initialized: now 1790422544479634 us; error 46 us; skew 500 ppm
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.483085 6181 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:35:44.484134 6181 webserver.cc:460] Webserver started at http://127.0.98.2:46239/ using document root <none> and password file <none>
I20260926 11:35:44.484776 6181 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:44.484876 6181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:44.485095 6181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:44.487279 6181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "d165778d5ae241e9940fa2f94ee6a6fd"
format_stamp: "Formatted at 2026-09-26 11:35:44 on dist-test-slave-4pw5"
server_key: "d6b8bd67d5f1bee2e744758bf80692e2"
server_key_iv: "5b284e14d93c7d1da9513c3417dafcba"
server_key_version: "encryptionkey@0"
I20260926 11:35:44.487931 6181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d165778d5ae241e9940fa2f94ee6a6fd"
format_stamp: "Formatted at 2026-09-26 11:35:44 on dist-test-slave-4pw5"
server_key: "d6b8bd67d5f1bee2e744758bf80692e2"
server_key_iv: "5b284e14d93c7d1da9513c3417dafcba"
server_key_version: "encryptionkey@0"
I20260926 11:35:44.492072 6181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:35:44.494874 6196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:44.496698 6181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:44.496905 6181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "d165778d5ae241e9940fa2f94ee6a6fd"
format_stamp: "Formatted at 2026-09-26 11:35:44 on dist-test-slave-4pw5"
server_key: "d6b8bd67d5f1bee2e744758bf80692e2"
server_key_iv: "5b284e14d93c7d1da9513c3417dafcba"
server_key_version: "encryptionkey@0"
I20260926 11:35:44.497062 6181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:44.525830 6181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:44.530247 6181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:44.530481 6181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:44.531134 6181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:44.532130 6181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:44.532202 6181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:44.532287 6181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:44.532337 6181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:44.541687 6181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35983
I20260926 11:35:44.541721 6309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35983 every 8 connection(s)
I20260926 11:35:44.542819 6181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:44.549288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 6181
I20260926 11:35:44.549451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:44.549803 392 external_mini_cluster.cc:1428] Setting key fc92974dffdb94c8cd6e5fa1d22cb8c8
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:35:44.559878 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:44.545208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37925 (local address 127.0.98.62:43081)
0926 11:35:44.545369 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:44.545373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:44.546479 (+ 1106us) server_negotiation.cc:407] Connection header received
0926 11:35:44.548040 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:44.548043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:44.548104 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:44.548204 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:44.549609 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.550361 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.551755 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.552005 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.552123 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:44.555248 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:44.555253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:44.555255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:44.555284 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:44.557216 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.558014 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.558018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.558023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.558126 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.558535 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.558539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.558541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.558766 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:44.558891 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:44.559516 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 11:35:44.559694 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:35:44.560889 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:44.545720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43081 (local address 127.0.98.2:37925)
0926 11:35:44.546211 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:44.546267 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:44.547803 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:44.548416 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:44.548429 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:44.548868 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:44.549382 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:44.549397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.550482 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.550485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:44.551553 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:44.551563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.552177 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.552181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:44.552541 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:44.553256 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:44.553274 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:44.555000 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:44.557400 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:44.557408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:44.557432 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:44.557890 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:44.558245 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:44.558248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:44.558250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:44.558427 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:44.558890 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:44.558900 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:44.559208 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 11:35:44.559458 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 11:35:44.559895 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":131,"threads_started":1}
I20260926 11:35:44.562317 6310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:35:44.562652 6310 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:44.563349 6310 heartbeater.cc:511] Master 127.0.98.62:43081 requested a full tablet report, sending...
I20260926 11:35:44.564764 5983 ts_manager.cc:194] Registered new tserver with Master: d165778d5ae241e9940fa2f94ee6a6fd (127.0.98.2:35983)
I20260926 11:35:44.565650 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:35:44.584635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:44.701742 6315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:44.702045 6315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:44.702091 6315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:44.706367 6315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:44.706460 6315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:44.706525 6315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:44.712174 6315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:44.713281 6315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:44.714329 6315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:44.724594 6321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:44.724594 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:44.724637 6315 server_base.cc:1034] running on GCE node
W20260926 11:35:44.724594 6320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:44.725394 6315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:44.726109 6315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:44.727331 6315 hybrid_clock.cc:648] HybridClock initialized: now 1790422544727311 us; error 33 us; skew 500 ppm
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.730361 6315 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:35:44.731349 6315 webserver.cc:460] Webserver started at http://127.0.98.3:44707/ using document root <none> and password file <none>
I20260926 11:35:44.732002 6315 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:44.732096 6315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:44.732336 6315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:44.734539 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "c52f6d5f1eb64c18b05c257bc9ac658b"
format_stamp: "Formatted at 2026-09-26 11:35:44 on dist-test-slave-4pw5"
server_key: "c49334670e38cd6fd7ab667eda0864fe"
server_key_iv: "a301bf4f392e4a1b8f4a5e5ce7d6a979"
server_key_version: "encryptionkey@0"
I20260926 11:35:44.735431 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c52f6d5f1eb64c18b05c257bc9ac658b"
format_stamp: "Formatted at 2026-09-26 11:35:44 on dist-test-slave-4pw5"
server_key: "c49334670e38cd6fd7ab667eda0864fe"
server_key_iv: "a301bf4f392e4a1b8f4a5e5ce7d6a979"
server_key_version: "encryptionkey@0"
I20260926 11:35:44.739828 6315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:44.742628 6330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:44.744270 6315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:44.744432 6315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "c52f6d5f1eb64c18b05c257bc9ac658b"
format_stamp: "Formatted at 2026-09-26 11:35:44 on dist-test-slave-4pw5"
server_key: "c49334670e38cd6fd7ab667eda0864fe"
server_key_iv: "a301bf4f392e4a1b8f4a5e5ce7d6a979"
server_key_version: "encryptionkey@0"
I20260926 11:35:44.744573 6315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:44.769197 6315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:44.773273 6315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:44.773540 6315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:44.774228 6315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:44.775198 6315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:44.775282 6315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:44.775364 6315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:44.775413 6315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:44.786176 6315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38475
I20260926 11:35:44.786197 6443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38475 every 8 connection(s)
I20260926 11:35:44.787187 6315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:44.792351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 6315
I20260926 11:35:44.792533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:44.792838 392 external_mini_cluster.cc:1428] Setting key eeb91e4d2412e745fd814c54f0224ed4
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:44.802420 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:44.789188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35813 (local address 127.0.98.62:43081)
0926 11:35:44.789366 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:44.789371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:44.790265 (+ 894us) server_negotiation.cc:407] Connection header received
0926 11:35:44.791744 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:44.791747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:44.791802 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:44.791890 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:44.793334 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.793925 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.795415 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.795733 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.795819 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:44.798607 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:44.798611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:44.798613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:44.798639 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:44.800187 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.800796 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.800801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.800804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.800888 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.801228 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.801231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.801232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.801412 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:44.801482 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:44.802190 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 11:35:44.802297 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 11:35:44.803495 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:44.789638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43081 (local address 127.0.98.3:35813)
0926 11:35:44.790099 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:44.790143 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:44.791413 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:44.792080 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:44.792087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:44.792468 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:44.793135 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:44.793174 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.794085 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.794088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:44.795298 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:44.795307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.795865 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.795868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:44.796015 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:44.796689 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:44.796706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:44.798405 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:44.800317 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:44.800322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:44.800349 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:44.800687 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:44.800980 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:44.800982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:44.800999 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:44.801148 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:44.801559 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:44.801566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:44.801844 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 11:35:44.802149 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 11:35:44.802583 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260926 11:35:44.805087 6444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43081
I20260926 11:35:44.805397 6444 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:44.806097 6444 heartbeater.cc:511] Master 127.0.98.62:43081 requested a full tablet report, sending...
I20260926 11:35:44.807467 5983 ts_manager.cc:194] Registered new tserver with Master: c52f6d5f1eb64c18b05c257bc9ac658b (127.0.98.3:38475)
I20260926 11:35:44.808142 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35813
I20260926 11:35:44.819350 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/test-admin.keytab.
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.837755 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 11:35:44.840718 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:44.839161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.0.98.1:36273)
0926 11:35:44.839436 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:44.839442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:44.839467 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:35:44.839517 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:44.839522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:44.839909 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:44.840036 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:44.840510 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44530: BlockingRecv error: recv got EOF from 127.0.0.1:44530 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
W20260926 11:35:44.840896 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.842028 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.843205 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:35:44.854600 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:44.844382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.0.98.1:36273)
0926 11:35:44.844559 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:44.844563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:44.844703 (+ 140us) server_negotiation.cc:407] Connection header received
0926 11:35:44.844886 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:44.844889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:44.844935 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:44.845024 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:44.845644 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.846410 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.847310 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.847796 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.847920 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:44.850715 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:44.850725 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:44.850740 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:44.850769 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:44.852656 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.853126 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.853132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.853133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.853227 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.853521 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.853524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.853525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.853914 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:44.854138 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:44.854261 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 11:35:44.854421 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 11:35:44.858912 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:44.857741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.0.98.1:36273)
0926 11:35:44.857885 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:44.857890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:44.858003 (+ 113us) server_negotiation.cc:407] Connection header received
0926 11:35:44.858152 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:44.858157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:44.858232 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:44.858320 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:44.858780 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44536: BlockingRecv error: recv got EOF from 127.0.0.1:44536 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 11:35:44.859077 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.860288 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.861152 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:35:44.870985 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:44.862034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.0.98.1:36273)
0926 11:35:44.862184 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:44.862191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:44.862284 (+ 93us) server_negotiation.cc:407] Connection header received
0926 11:35:44.862544 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:44.862548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:44.862595 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:44.862695 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:44.863298 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.863947 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.864831 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.865091 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.865213 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:44.867848 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:44.867852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:44.867854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:44.867885 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:44.869419 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.869825 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.869828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.869829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.869901 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.870190 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.870193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.870194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.870360 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:44.870471 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:44.870755 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 11:35:44.870859 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 11:35:44.874239 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:44.873120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.0.98.1:36273)
0926 11:35:44.873238 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:44.873243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:44.873382 (+ 139us) server_negotiation.cc:407] Connection header received
0926 11:35:44.873575 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:44.873578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:44.873617 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:44.873706 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:44.874120 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44540: BlockingRecv error: recv got EOF from 127.0.0.1:44540 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 11:35:44.874418 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422511662684-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.875474 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:44.876573 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:35:44.886978 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:44.877598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.0.98.1:36273)
0926 11:35:44.877740 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:44.877746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:44.877912 (+ 166us) server_negotiation.cc:407] Connection header received
0926 11:35:44.878127 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:44.878130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:44.878192 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:44.878291 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:44.878893 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.879477 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.880563 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:44.880897 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:44.881006 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:44.883690 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:44.883695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:44.883697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:44.883739 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:44.885229 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.885736 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.885740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.885742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.885828 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:44.886115 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:44.886119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:44.886121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:44.886309 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:44.886471 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:44.886703 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 11:35:44.886811 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:35:44.888094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 6047
I20260926 11:35:44.893687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 6181
I20260926 11:35:44.899928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 6315
I20260926 11:35:44.905387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 5953
2026-09-26T11:35:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2378 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[6464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[6464](info): set up 2 sockets
Sep 26 11:35:44 dist-test-slave-4pw5 krb5kdc[6464](info): commencing operation
krb5kdc: starting...
W20260926 11:35:46.000694 392 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:46 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 11:35:46.138895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43973
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:46.254776 6477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:46.255143 6477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:46.255214 6477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:46.259480 6477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:46.259634 6477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:46.259690 6477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:46.259753 6477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:46.259795 6477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:46.265717 6477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:46.267081 6477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:46.268246 6477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:46.277686 6485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:46.277668 6482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:46.277755 6477 server_base.cc:1034] running on GCE node
W20260926 11:35:46.277674 6483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:46.278692 6477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:46.279996 6477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:46.281193 6477 hybrid_clock.cc:648] HybridClock initialized: now 1790422546281156 us; error 54 us; skew 500 ppm
Sep 26 11:35:46 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:46.284387 6477 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 11:35:46.285966 6477 webserver.cc:460] Webserver started at http://127.0.98.62:33581/ using document root <none> and password file <none>
I20260926 11:35:46.286788 6477 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:46.286979 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:46.287454 6477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:46.290094 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "3de9bb59f0b048119f50c9f51ddca9a2"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "834acab20bd675ee0dcfc84842641487"
server_key_iv: "6a2736e324dfaa99210efa3e8f7c65b0"
server_key_version: "encryptionkey@0"
I20260926 11:35:46.290747 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "3de9bb59f0b048119f50c9f51ddca9a2"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "834acab20bd675ee0dcfc84842641487"
server_key_iv: "6a2736e324dfaa99210efa3e8f7c65b0"
server_key_version: "encryptionkey@0"
I20260926 11:35:46.295369 6477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 11:35:46.298728 6492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:46.300376 6477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:46.300540 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "3de9bb59f0b048119f50c9f51ddca9a2"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "834acab20bd675ee0dcfc84842641487"
server_key_iv: "6a2736e324dfaa99210efa3e8f7c65b0"
server_key_version: "encryptionkey@0"
I20260926 11:35:46.300693 6477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:46.351667 6477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:46.355870 6477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:46.356161 6477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:46.364668 6477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43973
I20260926 11:35:46.364969 6544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43973 every 8 connection(s)
I20260926 11:35:46.365842 6477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:46.367383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 6477
I20260926 11:35:46.367530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:46.367880 392 external_mini_cluster.cc:1428] Setting key a960e09821fc5fc427e5e262684e3ead
I20260926 11:35:46.369460 6545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:46.376606 6545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: Bootstrap starting.
Sep 26 11:35:46 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:46.379907 6545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:46.380736 6545 log.cc:824] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:46.383230 6545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: No bootstrap required, opened a new log
I20260926 11:35:46.385644 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:46.369379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.0.98.62:43973)
0926 11:35:46.369858 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:46.369875 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:46.369912 (+ 37us) server_negotiation.cc:407] Connection header received
0926 11:35:46.370757 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:46.370778 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:46.371152 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:46.371764 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:46.372721 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.373829 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.375131 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.375885 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.376036 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:46.379798 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:46.379806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:46.379825 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:46.379868 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:46.382101 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:46.382941 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:46.382946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:46.382952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:46.383095 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:46.383441 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:46.383444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:46.383446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:46.383972 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:46.384209 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:46.384517 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 11:35:46.384844 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":185,"threads_started":1}
I20260926 11:35:46.387333 6545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } }
I20260926 11:35:46.387653 6545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:46.387744 6545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de9bb59f0b048119f50c9f51ddca9a2, State: Initialized, Role: FOLLOWER
I20260926 11:35:46.388232 6545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } }
I20260926 11:35:46.388407 6545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:46.388495 6545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:46.388623 6545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:46.389658 6545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } }
I20260926 11:35:46.390070 6545 leader_election.cc:302] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de9bb59f0b048119f50c9f51ddca9a2; no voters:
I20260926 11:35:46.390491 6545 leader_election.cc:288] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:46.390631 6550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:46.390980 6550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 3de9bb59f0b048119f50c9f51ddca9a2, State: Running, Role: LEADER
I20260926 11:35:46.391392 6550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } }
I20260926 11:35:46.392297 6545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:46.393429 6551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } }
I20260926 11:35:46.393596 6551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:46.394006 6559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:46.394728 6552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de9bb59f0b048119f50c9f51ddca9a2. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } }
I20260926 11:35:46.394933 6552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:46.397259 6559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:46.404237 6559 catalog_manager.cc:1347] Generated new cluster ID: 631d8c3567f841c3ac5ccae91a0a4419
I20260926 11:35:46.404340 6559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:46.438129 6559 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:46.439424 6559 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:46.450516 6559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: Generated new TSK 0
I20260926 11:35:46.451458 6559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 11:35:46.496433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:46.622203 6571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:46.622572 6571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:46.622656 6571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:46.627029 6571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:46.627173 6571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:46.627251 6571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:46.634439 6571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:46.635721 6571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:46.636828 6571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:46.647543 6579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:46.647550 6576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:46.647543 6577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:46.647753 6571 server_base.cc:1034] running on GCE node
I20260926 11:35:46.648312 6571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:46.648967 6571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:46.650167 6571 hybrid_clock.cc:648] HybridClock initialized: now 1790422546650135 us; error 32 us; skew 500 ppm
Sep 26 11:35:46 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:46.653548 6571 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 11:35:46.654582 6571 webserver.cc:460] Webserver started at http://127.0.98.1:35399/ using document root <none> and password file <none>
I20260926 11:35:46.655388 6571 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:46.655483 6571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:46.655741 6571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:46.658015 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "4352e835c189494db1abdfadc82db152"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "aa3f1bcd3a83302e10252fa530600ef3"
server_key_iv: "280c4a357bb2f8fc41b7931a5429b027"
server_key_version: "encryptionkey@0"
I20260926 11:35:46.658595 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4352e835c189494db1abdfadc82db152"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "aa3f1bcd3a83302e10252fa530600ef3"
server_key_iv: "280c4a357bb2f8fc41b7931a5429b027"
server_key_version: "encryptionkey@0"
I20260926 11:35:46.662752 6571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 11:35:46.665717 6586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:46.667086 6571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 11:35:46.667268 6571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "4352e835c189494db1abdfadc82db152"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "aa3f1bcd3a83302e10252fa530600ef3"
server_key_iv: "280c4a357bb2f8fc41b7931a5429b027"
server_key_version: "encryptionkey@0"
I20260926 11:35:46.667403 6571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:46.708087 6571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:46.711800 6571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:46.712061 6571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:46.712750 6571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:46.713694 6571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:46.713745 6571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:46.713821 6571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:46.713855 6571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:46.724716 6571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34257
I20260926 11:35:46.724858 6699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34257 every 8 connection(s)
I20260926 11:35:46.725867 6571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:46.735093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 6571
I20260926 11:35:46.735248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:46.735566 392 external_mini_cluster.cc:1428] Setting key 801531e710a91a043a0f058f1a4a24d9
Sep 26 11:35:46 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:46.742393 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:46.728233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45801 (local address 127.0.98.62:43973)
0926 11:35:46.728384 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:46.728388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:46.729286 (+ 898us) server_negotiation.cc:407] Connection header received
0926 11:35:46.730688 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:46.730691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:46.730741 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:46.730822 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:46.732258 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.732908 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.734219 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.734481 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.734608 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:46.737670 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:46.737675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:46.737680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:46.737713 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:46.739556 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:46.740420 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:46.740423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:46.740425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:46.740551 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:46.741053 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:46.741056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:46.741057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:46.741291 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:46.741389 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:46.742137 (+ 748us) server_negotiation.cc:299] Negotiation successful
0926 11:35:46.742264 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:35:46.743877 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:46.728629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:45801)
0926 11:35:46.729104 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:46.729150 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:46.730439 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:46.730995 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:46.731003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:46.731382 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:46.732061 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:46.732077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.733077 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.733081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:46.734059 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:46.734071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.734754 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.734761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:46.735013 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:46.735856 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:46.735881 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:46.737468 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:46.739765 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:46.739775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:46.739796 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:46.740275 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:46.740678 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:46.740684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:46.740689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:46.740923 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:46.741421 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:46.741431 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:46.741822 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 11:35:46.742090 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 11:35:46.742512 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260926 11:35:46.745548 6700 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43973
I20260926 11:35:46.745837 6700 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:35:46.746451 6700 heartbeater.cc:511] Master 127.0.98.62:43973 requested a full tablet report, sending...
I20260926 11:35:46.748442 6509 ts_manager.cc:194] Registered new tserver with Master: 4352e835c189494db1abdfadc82db152 (127.0.98.1:34257)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260926 11:35:46.749924 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 11:35:46.768199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:46.890291 6705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:46.890594 6705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:46.890638 6705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:46.895138 6705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:46.895207 6705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:46.895313 6705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:46.901046 6705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:46.902331 6705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:46.903460 6705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:46.913918 6713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:46.913957 6711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:46.913957 6710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:46.914296 6705 server_base.cc:1034] running on GCE node
I20260926 11:35:46.914740 6705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:46.915441 6705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:46.916678 6705 hybrid_clock.cc:648] HybridClock initialized: now 1790422546916647 us; error 56 us; skew 500 ppm
Sep 26 11:35:46 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:46.919483 6705 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 11:35:46.920514 6705 webserver.cc:460] Webserver started at http://127.0.98.2:46201/ using document root <none> and password file <none>
I20260926 11:35:46.921064 6705 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:46.921123 6705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:46.921288 6705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:46.923404 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "6a4b1009d5894659b17fc8e09f3f7e98"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "d93c093fdcde6cb3600d704491e0d2a1"
server_key_iv: "159a349ec67e990146653bfe8b96c7b0"
server_key_version: "encryptionkey@0"
I20260926 11:35:46.924010 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6a4b1009d5894659b17fc8e09f3f7e98"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "d93c093fdcde6cb3600d704491e0d2a1"
server_key_iv: "159a349ec67e990146653bfe8b96c7b0"
server_key_version: "encryptionkey@0"
I20260926 11:35:46.927961 6705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:46.930557 6720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:46.932168 6705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:46.932283 6705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "6a4b1009d5894659b17fc8e09f3f7e98"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "d93c093fdcde6cb3600d704491e0d2a1"
server_key_iv: "159a349ec67e990146653bfe8b96c7b0"
server_key_version: "encryptionkey@0"
I20260926 11:35:46.932384 6705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:46.953061 6705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:46.956657 6705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:46.956826 6705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:46.957366 6705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:46.958212 6705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:46.958261 6705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:46.958303 6705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:46.958318 6705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:46.967852 6705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39887
I20260926 11:35:46.967867 6833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39887 every 8 connection(s)
I20260926 11:35:46.968842 6705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:46.975906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 6705
I20260926 11:35:46.976044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:46.976341 392 external_mini_cluster.cc:1428] Setting key f3162315f6f446994a275a6ebbcaf88b
Sep 26 11:35:46 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:46.983515 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:46.970737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56301 (local address 127.0.98.62:43973)
0926 11:35:46.970943 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:46.970948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:46.971929 (+ 981us) server_negotiation.cc:407] Connection header received
0926 11:35:46.973445 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:46.973448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:46.973510 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:46.973598 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:46.974878 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.975428 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.976805 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.977040 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.977157 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:46.979831 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:46.979834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:46.979837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:46.979863 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:46.981330 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:46.981899 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:46.981903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:46.981904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:46.981980 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:46.982344 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:46.982347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:46.982349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:46.982549 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:46.982665 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:46.983213 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 11:35:46.983345 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 11:35:46.984452 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:46.971212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:56301)
0926 11:35:46.971755 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:46.971799 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:46.973205 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:46.973787 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:46.973795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:46.974185 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:46.974711 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:46.974726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.975609 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.975612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:46.976648 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:46.976659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:46.977202 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:46.977206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:46.977385 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:46.978009 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:46.978026 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:46.979635 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:46.981463 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:46.981469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:46.981496 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:46.981774 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:46.982086 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:46.982089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:46.982092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:46.982242 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:46.982663 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:46.982671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:46.982949 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 11:35:46.983157 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 11:35:46.983527 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:35:46.985795 6834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43973
I20260926 11:35:46.986069 6834 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:46.986649 6834 heartbeater.cc:511] Master 127.0.98.62:43973 requested a full tablet report, sending...
I20260926 11:35:46.988026 6509 ts_manager.cc:194] Registered new tserver with Master: 6a4b1009d5894659b17fc8e09f3f7e98 (127.0.98.2:39887)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260926 11:35:46.988863 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 11:35:47.007838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:47.129012 6839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:47.129328 6839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:47.129379 6839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:47.133837 6839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:47.133935 6839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:47.134056 6839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:47.140378 6839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:47.141510 6839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:47.142529 6839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:47.153007 6844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:47.153039 6845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:47.153039 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:47.153299 6839 server_base.cc:1034] running on GCE node
I20260926 11:35:47.153707 6839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:47.154404 6839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:47.155694 6839 hybrid_clock.cc:648] HybridClock initialized: now 1790422547155675 us; error 47 us; skew 500 ppm
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:47.158826 6839 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 11:35:47.159890 6839 webserver.cc:460] Webserver started at http://127.0.98.3:36941/ using document root <none> and password file <none>
I20260926 11:35:47.160478 6839 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:47.160542 6839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:47.160723 6839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:47.162955 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "f8526fc109694c32a64b01e2ce5c2331"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "ad44e1ed3fc8d28268b4b7c1c23016e4"
server_key_iv: "f7e6ad3ea1f7b80e728260c185b2989d"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.163542 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f8526fc109694c32a64b01e2ce5c2331"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "ad44e1ed3fc8d28268b4b7c1c23016e4"
server_key_iv: "f7e6ad3ea1f7b80e728260c185b2989d"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.168146 6839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:47.170931 6854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:47.172511 6839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:47.172624 6839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "f8526fc109694c32a64b01e2ce5c2331"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "ad44e1ed3fc8d28268b4b7c1c23016e4"
server_key_iv: "f7e6ad3ea1f7b80e728260c185b2989d"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.172735 6839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:47.185374 6839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:47.189285 6839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:47.189459 6839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:47.190079 6839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:47.190989 6839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:47.191037 6839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:47.191079 6839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:47.191095 6839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:47.203293 6839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45343
I20260926 11:35:47.203311 6967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45343 every 8 connection(s)
I20260926 11:35:47.204502 6839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:47.205508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 6839
I20260926 11:35:47.205629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:47.205910 392 external_mini_cluster.cc:1428] Setting key 876ecbc715e2f8a8429e9debe81a3cce
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:47.220439 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.206545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47117 (local address 127.0.98.62:43973)
0926 11:35:47.208079 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.208083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.208099 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:35:47.209228 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.209231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.209277 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.209353 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.210604 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.211239 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.212525 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.212757 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.212846 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.215682 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.215686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.215689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.215719 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.217362 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.217955 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.217958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.217960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.218066 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.218787 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.218793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.218798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.219057 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.219192 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.220126 (+ 934us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.220273 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1451}
I20260926 11:35:47.221383 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.207053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:47117)
0926 11:35:47.207613 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.207709 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.209004 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.209494 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.209500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.209891 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.210422 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.210436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.211380 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.211382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.212404 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.212419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.212900 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.212902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.213046 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.213670 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.213686 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.215371 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.217511 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.217517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.217534 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.217845 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.218233 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.218236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.218237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.218403 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.219274 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.219313 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.219793 (+ 480us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.220059 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.220431 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":107,"threads_started":1}
I20260926 11:35:47.222640 6968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43973
I20260926 11:35:47.223033 6968 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:47.223743 6968 heartbeater.cc:511] Master 127.0.98.62:43973 requested a full tablet report, sending...
I20260926 11:35:47.224941 6509 ts_manager.cc:194] Registered new tserver with Master: f8526fc109694c32a64b01e2ce5c2331 (127.0.98.3:45343)
I20260926 11:35:47.225504 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47117
I20260926 11:35:47.233196 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 11:35:47.263230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41513
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43973,127.0.98.61:41513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:47.382157 6973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:47.382450 6973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:47.382493 6973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:47.386591 6973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:47.386665 6973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:47.386684 6973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:47.386701 6973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:47.386718 6973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:47.392468 6973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43973,127.0.98.61:41513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
W20260926 11:35:47.392792 6973 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43973,127.0.98.61:41513'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 11:35:47.393880 6973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:47.394975 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:35:47.404289 6973 server_base.cc:1034] running on GCE node
W20260926 11:35:47.404245 6979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:47.404266 6981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:47.404245 6978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:47.405292 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:47.406592 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:47.407799 6973 hybrid_clock.cc:648] HybridClock initialized: now 1790422547407748 us; error 74 us; skew 500 ppm
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:47.411019 6973 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 11:35:47.412104 6973 webserver.cc:460] Webserver started at http://127.0.98.61:33229/ using document root <none> and password file <none>
I20260926 11:35:47.412693 6973 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:47.412756 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:47.412930 6973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:47.415274 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data/instance:
uuid: "f9ef07cd30064f14abcfb957e065e9fc"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "5b95c0f216af6f360dd11d65797417b2"
server_key_iv: "8b34e6f9fc30c18bbc1b19ac392b078c"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.415967 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal/instance:
uuid: "f9ef07cd30064f14abcfb957e065e9fc"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "5b95c0f216af6f360dd11d65797417b2"
server_key_iv: "8b34e6f9fc30c18bbc1b19ac392b078c"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.420014 6973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:47.422439 6988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:47.424069 6973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:47.424247 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal
uuid: "f9ef07cd30064f14abcfb957e065e9fc"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "5b95c0f216af6f360dd11d65797417b2"
server_key_iv: "8b34e6f9fc30c18bbc1b19ac392b078c"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.424368 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:47.456015 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:47.459717 6973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:47.459955 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:47.468211 6973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41513
I20260926 11:35:47.468539 7040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41513 every 8 connection(s)
I20260926 11:35:47.469357 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:35:47.470799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 6973
I20260926 11:35:47.472954 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:47.477524 7041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:47.487293 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.472441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.0.98.61:41513)
0926 11:35:47.472891 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.472919 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.472963 (+ 44us) server_negotiation.cc:407] Connection header received
0926 11:35:47.473710 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.473728 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.474085 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.474385 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.475330 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.476755 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.478017 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.478537 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.478670 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.482037 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.482043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.482052 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.482079 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.484092 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.484571 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.484584 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.484589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.484700 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.485078 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.485082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.485084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.485601 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.485824 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.486096 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.486418 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":161,"threads_started":1}
I20260926 11:35:47.488991 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.479670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.0.98.62:43973)
0926 11:35:47.479820 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.479828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.480224 (+ 396us) server_negotiation.cc:407] Connection header received
0926 11:35:47.480364 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.480369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.480421 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.480503 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.481369 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.481944 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.482876 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.483096 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.483217 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.485468 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.485472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.485474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.485499 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.487070 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.487636 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.487640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.487653 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.487746 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.488022 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.488027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.488033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.488260 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.488382 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.488703 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.488829 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:35:47.489010 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.479648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:46956)
0926 11:35:47.480082 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.480131 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.480260 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.480556 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.480561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.481033 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.481240 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.481248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.482077 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.482080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.482750 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.482758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.483217 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.483221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.483336 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.483834 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.483846 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.485317 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.487185 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.487191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.487196 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.487477 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.487846 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.487848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.487849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.487917 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.488376 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.488382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.488493 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.488631 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.488829 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":169,"threads_started":1}
I20260926 11:35:47.490306 7041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 11:35:47.498495 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.490775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.0.1:41882)
0926 11:35:47.490894 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.490908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.491006 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.491222 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.491225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.491385 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.491464 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.491469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.492580 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.492585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.493091 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.493098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.493593 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.493597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.493749 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.494248 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.494259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.495539 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.497279 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.497281 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.497283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.497461 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.497709 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.497710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.497712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.497780 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.498098 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.498100 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.498179 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.498259 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.498364 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":36}
I20260926 11:35:47.498688 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.490896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.0.98.61:41513)
0926 11:35:47.491006 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.491010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.491021 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:35:47.491089 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.491093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.491132 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.491213 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.491604 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.492474 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.493242 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.493491 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.493596 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.495717 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.495720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.495722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.495742 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.497173 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.497557 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.497560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.497561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.497630 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.497853 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.497855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.497856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.498021 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.498092 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.498352 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.498515 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":53}
I20260926 11:35:47.501787 7041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: Bootstrap starting.
I20260926 11:35:47.504370 7041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:47.505184 7041 log.cc:824] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:47.507181 7041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: No bootstrap required, opened a new log
I20260926 11:35:47.510207 7041 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } }
I20260926 11:35:47.510438 7041 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:47.510473 7041 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ef07cd30064f14abcfb957e065e9fc, State: Initialized, Role: FOLLOWER
I20260926 11:35:47.510921 7041 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } }
I20260926 11:35:47.511469 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } } }
I20260926 11:35:47.511687 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:35:47.512226 7041 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: configured and running, proceeding with master startup.
W20260926 11:35:47.517472 7058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:35:47.517575 7058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:35:47.519474 6973 master_runner.cc:428] Detected that this master f9ef07cd30064f14abcfb957e065e9fc is joining an existing cluster
I20260926 11:35:47.519557 6973 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41513
I20260926 11:35:47.532181 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.521754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.0.98.62:43973)
0926 11:35:47.521906 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.521911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.522284 (+ 373us) server_negotiation.cc:407] Connection header received
0926 11:35:47.522513 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.522517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.522563 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.522638 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.523264 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.524078 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.526549 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.526939 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.527034 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.528577 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.528581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.528583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.528611 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.530084 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.530596 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.530600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.530601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.530846 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.531143 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.531146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.531148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.531350 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.531467 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.531863 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.531996 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:35:47.532199 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.521708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.0.1:41886)
0926 11:35:47.522159 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.522179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.522305 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.522561 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.522564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.522960 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.523163 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.523171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.524206 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.524210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.525250 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.525261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.525715 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.525718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.525843 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.526378 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.526399 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.526879 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.529051 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.529409 (+ 358us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.529412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.530433 (+ 1021us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.530859 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.530863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.530866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.530997 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.531655 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.531658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.531737 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.531840 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.532015 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":29,"thread_start_us":120,"threads_started":1}
I20260926 11:35:47.532634 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.522094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.0.98.61:41513)
0926 11:35:47.522191 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.522197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.522257 (+ 60us) server_negotiation.cc:407] Connection header received
0926 11:35:47.522404 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.522409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.522447 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.522512 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.523297 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.524078 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.525370 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.525567 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.525703 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.527018 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.527022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.527026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.527052 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.528950 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.530578 (+ 1628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.530581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.530583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.530729 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.531100 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.531104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.531106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.531338 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.531483 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.531908 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.532109 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":30}
I20260926 11:35:47.532811 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.521836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:46960)
0926 11:35:47.522191 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.522208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.522352 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.522653 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.522656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.522984 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.523139 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.523147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.525405 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.525409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.526405 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.526415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.527147 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.527150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.527221 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.527789 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.527810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.528369 (+ 559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.530220 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.530223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.530225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.530486 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.530974 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.530977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.530978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.531065 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.531463 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.531466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.531657 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.532376 (+ 719us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.532550 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":17,"thread_start_us":113,"threads_started":1}
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 11:35:47.550275 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.537983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.0.98.61:41513)
0926 11:35:47.538122 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.538127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.538496 (+ 369us) server_negotiation.cc:407] Connection header received
0926 11:35:47.538721 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.538724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.538771 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.538849 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.539949 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.540583 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.541518 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.541734 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.541860 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.546408 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.546412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.546415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.546453 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.548131 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.548727 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.548732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.548736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.548829 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.549130 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.549135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.549137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.549443 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.549542 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.549970 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.550102 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:35:47.550321 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.537930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.0.1:41900)
0926 11:35:47.538349 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.538400 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.538552 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.538907 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.538914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.539643 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.539795 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.539805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.540727 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.540736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.541382 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.541392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.541869 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.541873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.541998 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.542605 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.542617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.546177 (+ 3560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.548290 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.548305 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.548314 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.548610 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.548933 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.548936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.548938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.549033 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.549603 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.549619 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.549744 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.549914 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.550162 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 11:35:47.552754 6509 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41513: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41513
}
attrs {
promote: true
}
}
dest_uuid: "3de9bb59f0b048119f50c9f51ddca9a2"
cas_config_opid_index: -1
I20260926 11:35:47.553515 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } }
I20260926 11:35:47.555657 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f9ef07cd30064f14abcfb957e065e9fc (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } } }
I20260926 11:35:47.556559 7015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260926 11:35:47.558979 7015 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Refusing update from remote peer 3de9bb59f0b048119f50c9f51ddca9a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 11:35:47.559782 7069 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41513
I20260926 11:35:47.560161 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } } }
I20260926 11:35:47.560416 6973 master.cc:537] Master@127.0.98.61:41513 shutting down...
I20260926 11:35:47.560406 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:47.560708 7069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 11:35:47.563220 6493 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 11:35:47.564270 6493 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 -> Peer f9ef07cd30064f14abcfb957e065e9fc (127.0.98.61:41513): Couldn't send request to peer f9ef07cd30064f14abcfb957e065e9fc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 11:35:47.565222 6973 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:47.565712 6973 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:47.565841 6973 tablet_replica.cc:330] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: stopping tablet replica
I20260926 11:35:47.581209 6973 master.cc:552] Master@127.0.98.61:41513 shutdown complete.
I20260926 11:35:47.581284 6973 master_runner.cc:305] Clearing existing system tablet
I20260926 11:35:47.581576 6973 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:47.581640 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:47.583217 6973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 11:35:47.584398 7076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:47.584699 6973 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:47.584837 6973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 11:35:47.584875 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal
uuid: "f9ef07cd30064f14abcfb957e065e9fc"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "5b95c0f216af6f360dd11d65797417b2"
server_key_iv: "8b34e6f9fc30c18bbc1b19ac392b078c"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.587438 6973 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 11:35:47.590592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41515
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43973,127.0.98.61:41513,127.0.98.60:41515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:35:47.590989 6973 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 11:35:47.591113 6973 log.cc:1196] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 11:35:47.591434 6973 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: Deleting consensus metadata
I20260926 11:35:47.591691 6973 master_runner.cc:315] Copying system tablet from 127.0.98.62:43973
I20260926 11:35:47.593169 6973 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43973
I20260926 11:35:47.604559 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.593815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:46962)
0926 11:35:47.594137 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.594154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.594274 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.594745 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.594749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.595005 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.595148 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.595157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.596449 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.596453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.597506 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.597520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.598252 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.598257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.598361 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.599024 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.599052 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.599533 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.602552 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.602556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.602559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.602852 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.603181 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.603184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.603186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.603283 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.603847 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.603850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.604011 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.604131 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.604322 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":137,"threads_started":1}
I20260926 11:35:47.604592 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.594273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.0.98.62:43973)
0926 11:35:47.594420 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.594427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.594445 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:35:47.594499 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.594504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.594566 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.594663 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.595312 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.596308 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.597666 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.598105 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.598240 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.600342 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.600348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.600354 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.600391 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.602411 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.602972 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.602976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.602978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.603074 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.603416 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.603420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.603423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.603732 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.603851 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.604248 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.604430 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:35:47.605263 6529 tablet_copy_service.cc:140] P 3de9bb59f0b048119f50c9f51ddca9a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f9ef07cd30064f14abcfb957e065e9fc ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46962)
I20260926 11:35:47.605453 6529 tablet_copy_service.cc:161] P 3de9bb59f0b048119f50c9f51ddca9a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f9ef07cd30064f14abcfb957e065e9fc at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46962: session id = f9ef07cd30064f14abcfb957e065e9fc-00000000000000000000000000000000
I20260926 11:35:47.607357 6529 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 11:35:47.609501 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 11:35:47.612365 6973 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: tablet copy: Starting download of 0 data blocks...
I20260926 11:35:47.612619 6973 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: tablet copy: Starting download of 1 WAL segments...
I20260926 11:35:47.614960 6973 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 11:35:47.617239 6529 tablet_copy_service.cc:342] P 3de9bb59f0b048119f50c9f51ddca9a2: Request end of tablet copy session f9ef07cd30064f14abcfb957e065e9fc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46962
I20260926 11:35:47.617391 6529 tablet_copy_service.cc:433] P 3de9bb59f0b048119f50c9f51ddca9a2: ending tablet copy session f9ef07cd30064f14abcfb957e065e9fc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f9ef07cd30064f14abcfb957e065e9fc
I20260926 11:35:47.626888 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:47.628938 7087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:47.629050 6973 server_base.cc:1034] running on GCE node
W20260926 11:35:47.629065 7084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:47.629462 7085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:47.629729 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:47.630050 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:47.631201 6973 hybrid_clock.cc:648] HybridClock initialized: now 1790422547631188 us; error 31 us; skew 500 ppm
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:47.632962 6973 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 11:35:47.633469 6973 webserver.cc:460] Webserver started at http://127.0.98.61:38295/ using document root <none> and password file <none>
I20260926 11:35:47.633679 6973 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:47.633724 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:47.635124 6973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:47.636220 7094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:47.636457 6973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 11:35:47.636572 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal
uuid: "f9ef07cd30064f14abcfb957e065e9fc"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "5b95c0f216af6f360dd11d65797417b2"
server_key_iv: "8b34e6f9fc30c18bbc1b19ac392b078c"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.636890 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:47.647696 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:47.650061 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:47.655738 6973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41513
I20260926 11:35:47.655824 7146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41513 every 8 connection(s)
I20260926 11:35:47.656531 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:35:47.658993 7147 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:35:47.660307 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: Bootstrap starting.
I20260926 11:35:47.671862 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:35:47.672256 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: Bootstrap complete.
I20260926 11:35:47.672946 7147 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } }
I20260926 11:35:47.673118 7147 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f9ef07cd30064f14abcfb957e065e9fc, State: Initialized, Role: LEARNER
I20260926 11:35:47.673240 7147 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } }
I20260926 11:35:47.673648 7150 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } } }
I20260926 11:35:47.673764 7147 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:47.673842 7150 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: This master's current role is: LEARNER
I20260926 11:35:47.677677 7161 catalog_manager.cc:1259] Loaded cluster ID: 631d8c3567f841c3ac5ccae91a0a4419
I20260926 11:35:47.677742 7161 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: loading cluster ID for follower catalog manager: success
I20260926 11:35:47.679855 7161 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: acquiring CA information for follower catalog manager: success
I20260926 11:35:47.681309 7161 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 11:35:47.736107 7077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:47.736472 7077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:47.736577 7077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:47.741179 7077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:47.741300 7077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:47.741336 7077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:47.741379 7077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:47.741410 7077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:47.747300 7077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43973,127.0.98.61:41513,127.0.98.60:41515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:47.748605 7077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:47.749768 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:35:47.753654 6700 heartbeater.cc:503] Master 127.0.98.62:43973 was elected leader, sending a full tablet report...
W20260926 11:35:47.760738 7166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:47.760792 7169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:47.761001 7077 server_base.cc:1034] running on GCE node
W20260926 11:35:47.760741 7167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:47.761904 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:47.763398 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:47.764575 7077 hybrid_clock.cc:648] HybridClock initialized: now 1790422547764554 us; error 36 us; skew 500 ppm
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:47.767925 7077 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 11:35:47.768918 7077 webserver.cc:460] Webserver started at http://127.0.98.60:46095/ using document root <none> and password file <none>
I20260926 11:35:47.769552 7077 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:47.769623 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:47.769850 7077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:47.772207 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data/instance:
uuid: "3865d02fa61c4cfea5debb15f877ed27"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "e586fe0b01bc1725d8ca3dd7a335ec26"
server_key_iv: "f4302ae09eacbcd9e9a3aba34369e884"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.772836 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal/instance:
uuid: "3865d02fa61c4cfea5debb15f877ed27"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "e586fe0b01bc1725d8ca3dd7a335ec26"
server_key_iv: "f4302ae09eacbcd9e9a3aba34369e884"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.777670 7077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:35:47.780529 7176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:47.782053 7077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:47.782279 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal
uuid: "3865d02fa61c4cfea5debb15f877ed27"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "e586fe0b01bc1725d8ca3dd7a335ec26"
server_key_iv: "f4302ae09eacbcd9e9a3aba34369e884"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.782437 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:47.804247 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:47.808009 7077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:47.808239 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:47.817341 7077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41515
I20260926 11:35:47.817718 7228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41515 every 8 connection(s)
I20260926 11:35:47.818476 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
I20260926 11:35:47.820015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 7077
I20260926 11:35:47.822083 7229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:47.826226 7229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:47.838276 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.821638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.0.98.60:41515)
0926 11:35:47.822175 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.822228 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.822289 (+ 61us) server_negotiation.cc:407] Connection header received
0926 11:35:47.823093 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.823114 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.823531 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.824015 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.824998 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.826136 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.827507 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.828413 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.828585 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.831510 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.831517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.831529 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.831566 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.834184 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.834851 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.834857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.834863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.835027 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.835418 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.835422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.835423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.836004 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.836254 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.836648 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.837016 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":151,"threads_started":1}
I20260926 11:35:47.840308 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.828730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.0.98.62:43973)
0926 11:35:47.828886 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.828893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.829483 (+ 590us) server_negotiation.cc:407] Connection header received
0926 11:35:47.829592 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.829597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.829660 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.829766 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.830469 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.831037 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.832296 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.832607 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.832742 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.836014 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.836019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.836022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.836058 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.838136 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.838819 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.838823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.838825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.838933 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.839229 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.839233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.839234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.839509 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.839657 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.840067 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.840186 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:35:47.840399 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.828476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:46964)
0926 11:35:47.829337 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.829392 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.829501 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.829822 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.829826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.830200 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.830330 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.830336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.831184 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.831187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.832148 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.832164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.832779 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.832787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.832994 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.833541 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.833558 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.835835 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.838277 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.838286 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.838291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.838702 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.839042 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.839044 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.839046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.839126 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.839683 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.839689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.839818 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.839977 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.840196 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":189,"threads_started":1}
I20260926 11:35:47.841513 7229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 11:35:47.852579 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.842214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.0.1:41914)
0926 11:35:47.842366 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.842382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.842475 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.842946 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.842949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.843417 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.843549 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.843558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.844699 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.844704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.845478 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.845489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.846077 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.846081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.846166 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.846825 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.846838 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.848413 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.850454 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.850457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.850459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.850719 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.851119 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.851130 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.851131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.851384 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.852054 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.852058 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.852156 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.852258 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.852389 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 11:35:47.852645 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.842323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.0.98.61:41513)
0926 11:35:47.842668 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.842673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.842692 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:35:47.842739 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.842741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.842780 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.842864 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.843816 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.844557 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.845653 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.845899 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.846019 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.848603 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.848606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.848609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.848672 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.850319 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.850846 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.850850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.850851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.850953 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.851707 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.851710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.851712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.851938 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.852030 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.852361 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.852516 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260926 11:35:47.853609 7229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 11:35:47.863067 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.854220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.0.1:50378)
0926 11:35:47.854383 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.854396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.854539 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.854772 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.854775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.854962 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.855031 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.855037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.856269 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.856272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.856803 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.856812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.857286 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.857288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.857353 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.857935 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.857947 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.859377 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.861405 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.861408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.861410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.861691 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.862044 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.862047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.862048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.862133 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.862584 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.862588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.862680 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.862794 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.862924 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 11:35:47.863116 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.854425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.0.98.60:41515)
0926 11:35:47.854539 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.854544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.854557 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:35:47.854633 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.854636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.854678 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.854764 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.855139 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.856110 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.856932 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.857166 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.857255 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.859550 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.859553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.859555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.859638 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.861275 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.861812 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.861815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.861816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.861893 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.862264 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.862267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.862268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.862460 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.862546 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.862869 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.863003 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":52}
I20260926 11:35:47.866398 7229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Bootstrap starting.
I20260926 11:35:47.868804 7229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:47.869515 7229 log.cc:824] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:47.871675 7229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: No bootstrap required, opened a new log
I20260926 11:35:47.876044 7229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } }
I20260926 11:35:47.876318 7229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:47.876395 7229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3865d02fa61c4cfea5debb15f877ed27, State: Initialized, Role: FOLLOWER
I20260926 11:35:47.876968 7229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } }
I20260926 11:35:47.877569 7235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } } }
I20260926 11:35:47.877732 7235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:35:47.878356 7229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 11:35:47.883651 7246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:35:47.883760 7246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:35:47.887483 7077 master_runner.cc:428] Detected that this master 3865d02fa61c4cfea5debb15f877ed27 is joining an existing cluster
I20260926 11:35:47.887620 7077 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:41515
I20260926 11:35:47.892371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 6571
I20260926 11:35:47.902181 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.890548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.0.98.61:41513)
0926 11:35:47.890693 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.890697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.891614 (+ 917us) server_negotiation.cc:407] Connection header received
0926 11:35:47.891812 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.891815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.891868 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.891920 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.892548 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.893213 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.894483 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.894846 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.894925 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.897813 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.897816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.897819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.897851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.900358 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.900870 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.900874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.900876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.900963 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.901248 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.901251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.901253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.901458 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.901526 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.901870 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.902007 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:35:47.900914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34257
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43973
--webserver_port=35399
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43973,127.0.98.61:41513
--tserver_master_addrs=127.0.98.62:43973,127.0.98.61:41513,127.0.98.60:41515 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:35:47.902679 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.890221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.0.1:41926)
0926 11:35:47.891484 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.891497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.891691 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.892055 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.892058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.892292 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.892411 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.892418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.893388 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.893391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.894313 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.894323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.895101 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.895105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.895196 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.896453 (+ 1257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.896468 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.896927 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.900486 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.900490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.900492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.900761 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.901068 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.901071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.901073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.901152 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.901595 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.901598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.901697 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.902337 (+ 640us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.902488 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"mutex_wait_us":813,"thread_start_us":101,"threads_started":1}
I20260926 11:35:47.908581 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.890827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.0.1:50382)
0926 11:35:47.896278 (+ 5451us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.896295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.897057 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.897341 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.897344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.897540 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.897660 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.897667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.899444 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.899447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.901255 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.901266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.901959 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.901962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.902082 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.903942 (+ 1860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.903964 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.904531 (+ 567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.906848 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.906851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.906854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.907105 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.907435 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.907438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.907440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.907548 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.908054 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.908057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.908154 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.908252 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.908400 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":106,"threads_started":1}
I20260926 11:35:47.909152 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.890865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.0.98.60:41515)
0926 11:35:47.890973 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.890977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.896359 (+ 5382us) server_negotiation.cc:407] Connection header received
0926 11:35:47.897147 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.897150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.897185 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.897229 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.898180 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.898934 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.901392 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.901711 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.901792 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.904697 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.904705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.904710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.904771 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.906554 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.907259 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.907262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.907263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.907342 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.907676 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.907679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.907681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.907945 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.908034 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.908323 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.908468 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":47}
I20260926 11:35:47.910256 7235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:47.910454 7235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } }
I20260926 11:35:47.912663 7235 leader_election.cc:288] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973), f9ef07cd30064f14abcfb957e065e9fc (127.0.98.61:41513)
I20260926 11:35:47.913280 7121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3865d02fa61c4cfea5debb15f877ed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9ef07cd30064f14abcfb957e065e9fc" is_pre_election: true
I20260926 11:35:47.913268 6519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3865d02fa61c4cfea5debb15f877ed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" is_pre_election: true
I20260926 11:35:47.913460 7121 raft_consensus.cc:1799] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 LEARNER]: Handling vote request from an unknown peer 3865d02fa61c4cfea5debb15f877ed27
I20260926 11:35:47.913581 7121 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 3865d02fa61c4cfea5debb15f877ed27 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260926 11:35:47.914151 7177 leader_election.cc:302] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3865d02fa61c4cfea5debb15f877ed27; no voters: 3de9bb59f0b048119f50c9f51ddca9a2, f9ef07cd30064f14abcfb957e065e9fc
I20260926 11:35:47.914634 7235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:47.916177 7235 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 11:35:47.920696 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.890341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.0.98.62:43973)
0926 11:35:47.890494 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.890499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.891224 (+ 725us) server_negotiation.cc:407] Connection header received
0926 11:35:47.891466 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.891470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.891542 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.891630 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.892414 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.893433 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.895044 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.895930 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.896034 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.916976 (+ 20942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.916981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.916984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.917017 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.918797 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.919365 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.919369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.919371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.919461 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.919783 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.919787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.919788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.920032 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.920142 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.920415 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.920539 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:35:47.920701 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.890221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:46972)
0926 11:35:47.891124 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.891143 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.891318 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.891791 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.891794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.892054 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.892274 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.892284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.893640 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.893646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.894896 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.894911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.896221 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.896228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.896373 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.916173 (+ 19800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.916196 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.916764 (+ 568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.918940 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.918944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.918947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.919266 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.919561 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.919564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.919566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.919686 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.920129 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.920132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.920238 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.920341 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.920498 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":145,"threads_started":1}
Sep 26 11:35:47 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 11:35:47.934772 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.924912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.0.1:50396)
0926 11:35:47.925091 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.925105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.925183 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.925468 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.925471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.925675 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.925784 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.925790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.926851 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.926855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.927832 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.927845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.928450 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.928455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.928562 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.929133 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.929150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.930748 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.932856 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.932860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.932862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.933106 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.933658 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.933663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.933666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.933760 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.934215 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.934218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.934322 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.934407 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.934585 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 11:35:47.935360 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.924981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.0.98.60:41515)
0926 11:35:47.925088 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.925093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.925145 (+ 52us) server_negotiation.cc:407] Connection header received
0926 11:35:47.925301 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.925304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.925357 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.925441 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.925905 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.926676 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.927975 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.928327 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.928490 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.930905 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.930909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.930912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.930937 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.932723 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.933429 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.933432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.933434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.933546 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.933874 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.933877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.933879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.934104 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.935002 (+ 898us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.935083 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.935210 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 11:35:47.936434 6509 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:41515: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 41515
}
attrs {
promote: true
}
}
dest_uuid: "3de9bb59f0b048119f50c9f51ddca9a2"
cas_config_opid_index: 5
I20260926 11:35:47.937127 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } }
I20260926 11:35:47.938568 7071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3865d02fa61c4cfea5debb15f877ed27 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.940610 7071 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:41515
I20260926 11:35:47.940613 7203 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Refusing update from remote peer 3de9bb59f0b048119f50c9f51ddca9a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 11:35:47.940826 7071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.940922 7071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:47.941461 7069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:47.942214 7077 master.cc:537] Master@127.0.98.60:41515 shutting down...
W20260926 11:35:47.943243 6495 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 -> Peer 3865d02fa61c4cfea5debb15f877ed27 (127.0.98.60:41515): Couldn't send request to peer 3865d02fa61c4cfea5debb15f877ed27. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 11:35:47.947271 7077 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:47.947628 7077 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:47.947728 7077 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: stopping tablet replica
I20260926 11:35:47.949910 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.938520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.0.98.61:41513)
0926 11:35:47.939718 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.939723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.944931 (+ 5208us) server_negotiation.cc:407] Connection header received
0926 11:35:47.945184 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.945188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.945253 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.945324 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:47.945874 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.946670 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.948404 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.949026 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.949121 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.949306 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.949686 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.949773 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100}
I20260926 11:35:47.950599 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.938420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.0.1:41930)
0926 11:35:47.944834 (+ 6414us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.944853 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.945022 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.945598 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.945604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.945629 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:47.945732 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.945738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.946940 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.946943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.948253 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.948266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.949413 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.949416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.949516 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.949561 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.950371 (+ 810us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.950463 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6317}
I20260926 11:35:47.951172 7121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 LEARNER]: Refusing update from remote peer 3de9bb59f0b048119f50c9f51ddca9a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 11:35:47.951690 7069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 11:35:47.953794 7121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3865d02fa61c4cfea5debb15f877ed27 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.956626 7150 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de9bb59f0b048119f50c9f51ddca9a2. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.956779 7150 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: This master's current role is: LEARNER
I20260926 11:35:47.956880 7150 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: true } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.956954 7150 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: This master's current role is: LEARNER
I20260926 11:35:47.957858 7069 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: attempting to promote NON_VOTER f9ef07cd30064f14abcfb957e065e9fc to VOTER
I20260926 11:35:47.958360 7069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } }
I20260926 11:35:47.959827 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.959487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.0.1:50400)
0926 11:35:47.959673 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.959745 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 11:35:47.959782 7121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 LEARNER]: Refusing update from remote peer 3de9bb59f0b048119f50c9f51ddca9a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 11:35:47.960536 7071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 11:35:47.960778 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.960314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.0.1:50416)
0926 11:35:47.960576 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.960688 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
W20260926 11:35:47.961053 6495 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 -> Peer 3865d02fa61c4cfea5debb15f877ed27 (127.0.98.60:41515): Couldn't send request to peer 3865d02fa61c4cfea5debb15f877ed27. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 11:35:47.962529 7070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f9ef07cd30064f14abcfb957e065e9fc (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.962973 7121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f9ef07cd30064f14abcfb957e065e9fc (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.964931 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.965061 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:47.965193 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.965266 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:47.965634 7077 master.cc:552] Master@127.0.98.60:41515 shutdown complete.
I20260926 11:35:47.965692 7077 master_runner.cc:305] Clearing existing system tablet
I20260926 11:35:47.965955 7077 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:47.966017 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:47.967842 7077 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 11:35:47.967928 7150 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:47.968073 7150 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:35:47.969048 7260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:47.969277 7077 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:47.969594 7077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:47.969695 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal
uuid: "3865d02fa61c4cfea5debb15f877ed27"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "e586fe0b01bc1725d8ca3dd7a335ec26"
server_key_iv: "f4302ae09eacbcd9e9a3aba34369e884"
server_key_version: "encryptionkey@0"
I20260926 11:35:47.972431 7077 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:47.975481 7077 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 11:35:47.975754 7077 log.cc:1196] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 11:35:47.976166 7077 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Deleting consensus metadata
I20260926 11:35:47.976548 7077 master_runner.cc:315] Copying system tablet from 127.0.98.62:43973
I20260926 11:35:47.977854 7077 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43973
I20260926 11:35:47.987795 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.978706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.0.98.62:43973)
0926 11:35:47.978845 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:47.978850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:47.979069 (+ 219us) server_negotiation.cc:407] Connection header received
0926 11:35:47.979269 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:47.979272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:47.979331 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:47.979459 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:47.980072 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.980893 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.982104 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.982424 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.982538 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.984000 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:47.984004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:47.984007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:47.984036 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:47.985913 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.986447 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.986450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.986453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.986548 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:47.986847 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:47.986850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:47.986852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:47.987079 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:47.987180 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:47.987461 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 11:35:47.987628 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:35:47.987864 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:47.978593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:46974)
0926 11:35:47.978958 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:47.978972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:47.979131 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:47.979512 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:47.979515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:47.979783 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:47.979921 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.979929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.981021 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.981025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.981951 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:47.981963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:47.982571 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:47.982575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:47.982676 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:47.983306 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:47.983322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:47.983831 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:47.986045 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.986049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.986051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.986335 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.986652 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:47.986655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:47.986657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:47.986751 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:47.987177 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:47.987180 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:47.987286 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 11:35:47.987395 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 11:35:47.987624 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":98,"threads_started":1}
I20260926 11:35:47.988689 6529 tablet_copy_service.cc:140] P 3de9bb59f0b048119f50c9f51ddca9a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3865d02fa61c4cfea5debb15f877ed27 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46974)
I20260926 11:35:47.988787 6529 tablet_copy_service.cc:161] P 3de9bb59f0b048119f50c9f51ddca9a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3865d02fa61c4cfea5debb15f877ed27 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46974: session id = 3865d02fa61c4cfea5debb15f877ed27-00000000000000000000000000000000
I20260926 11:35:47.989912 6529 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 11:35:47.991402 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 11:35:47.994323 7077 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: tablet copy: Starting download of 0 data blocks...
I20260926 11:35:47.994706 7077 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: tablet copy: Starting download of 1 WAL segments...
I20260926 11:35:47.996125 6834 heartbeater.cc:503] Master 127.0.98.62:43973 was elected leader, sending a full tablet report...
I20260926 11:35:47.997611 7077 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 11:35:47.999788 6529 tablet_copy_service.cc:342] P 3de9bb59f0b048119f50c9f51ddca9a2: Request end of tablet copy session 3865d02fa61c4cfea5debb15f877ed27-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46974
I20260926 11:35:47.999897 6529 tablet_copy_service.cc:433] P 3de9bb59f0b048119f50c9f51ddca9a2: ending tablet copy session 3865d02fa61c4cfea5debb15f877ed27-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3865d02fa61c4cfea5debb15f877ed27
I20260926 11:35:48.008220 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:48.010357 7267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:48.010596 7270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:48.010442 7268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:48.011762 7077 server_base.cc:1034] running on GCE node
I20260926 11:35:48.011973 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:48.012317 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:48.013484 7077 hybrid_clock.cc:648] HybridClock initialized: now 1790422548013468 us; error 36 us; skew 500 ppm
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:48.015257 7077 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 11:35:48.015818 7077 webserver.cc:460] Webserver started at http://127.0.98.60:41429/ using document root <none> and password file <none>
I20260926 11:35:48.016033 7077 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:48.016079 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:48.017499 7077 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:48.018561 7277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.018838 7077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:48.018918 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal
uuid: "3865d02fa61c4cfea5debb15f877ed27"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "e586fe0b01bc1725d8ca3dd7a335ec26"
server_key_iv: "f4302ae09eacbcd9e9a3aba34369e884"
server_key_version: "encryptionkey@0"
I20260926 11:35:48.019016 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:48.053524 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:48.056046 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:48.061425 7077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41515
I20260926 11:35:48.061601 7332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41515 every 8 connection(s)
I20260926 11:35:48.062040 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
W20260926 11:35:48.063145 7255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:48.063509 7255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:48.063604 7255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 11:35:48.063872 7333 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:35:48.064932 7333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Bootstrap starting.
W20260926 11:35:48.069734 7255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:48.069814 7255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:48.069900 7255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:48.076521 7333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:35:48.076864 7333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Bootstrap complete.
I20260926 11:35:48.077474 7333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } }
I20260926 11:35:48.077616 7333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3865d02fa61c4cfea5debb15f877ed27, State: Initialized, Role: LEARNER
I20260926 11:35:48.077737 7333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } }
I20260926 11:35:48.077911 7255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35399
--tserver_master_addrs=127.0.98.62:43973,127.0.98.61:41513,127.0.98.60:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:48.078213 7333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:48.078517 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:48.078703 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: This master's current role is: LEARNER
I20260926 11:35:48.079308 7255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:48.080900 7347 catalog_manager.cc:1259] Loaded cluster ID: 631d8c3567f841c3ac5ccae91a0a4419
I20260926 11:35:48.081012 7347 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: loading cluster ID for follower catalog manager: success
I20260926 11:35:48.081508 7255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:35:48.083353 7347 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: acquiring CA information for follower catalog manager: success
I20260926 11:35:48.084817 7347 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 11:35:48.092123 7352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:48.092204 7349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:48.092226 7350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:48.092854 7255 server_base.cc:1034] running on GCE node
I20260926 11:35:48.093384 7255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:48.094218 7255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:48.095430 7255 hybrid_clock.cc:648] HybridClock initialized: now 1790422548095419 us; error 32 us; skew 500 ppm
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:48.098482 7255 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 11:35:48.099459 7255 webserver.cc:460] Webserver started at http://127.0.98.1:35399/ using document root <none> and password file <none>
I20260926 11:35:48.100107 7255 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:48.100227 7255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:48.104503 7255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:35:48.107120 7359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.108635 7255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:48.108780 7255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "4352e835c189494db1abdfadc82db152"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "aa3f1bcd3a83302e10252fa530600ef3"
server_key_iv: "280c4a357bb2f8fc41b7931a5429b027"
server_key_version: "encryptionkey@0"
I20260926 11:35:48.109388 7255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:48.120525 7255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:48.124109 7255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:48.124337 7255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:48.124963 7255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:48.125867 7255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:48.125916 7255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.125954 7255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:48.125994 7255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.136966 7255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34257
I20260926 11:35:48.137316 7472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34257 every 8 connection(s)
I20260926 11:35:48.138218 7255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:48.143079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 7255
I20260926 11:35:48.143283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 6705
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:48.151430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39887
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43973
--webserver_port=46201
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43973,127.0.98.61:41513
--tserver_master_addrs=127.0.98.62:43973,127.0.98.61:41513,127.0.98.60:41515 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 11:35:48.158591 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.140091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60101 (local address 127.0.98.61:41513)
0926 11:35:48.140228 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.140233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.141253 (+ 1020us) server_negotiation.cc:407] Connection header received
0926 11:35:48.142356 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.142359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.142418 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.142495 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.144982 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.145726 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.147846 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.148189 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.148274 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.153312 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.153318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.153320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.153351 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.155509 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.156415 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.156419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.156421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.156675 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.157111 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.157115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.157116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.157372 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.157456 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.158269 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.158406 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:35:48.158591 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.142050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39097 (local address 127.0.98.62:43973)
0926 11:35:48.142232 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.142237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.142254 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:35:48.142620 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.142623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.142669 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.142750 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.144667 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.145505 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.147846 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.148208 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.148278 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.153365 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.153369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.153371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.153403 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.155778 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.156415 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.156419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.156421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.156529 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.157111 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.157114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.157116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.157372 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.157456 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.158272 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.158406 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 11:35:48.158787 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.140103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39911 (local address 127.0.98.60:41515)
0926 11:35:48.140367 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.140378 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.141254 (+ 876us) server_negotiation.cc:407] Connection header received
0926 11:35:48.142499 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.142501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.142550 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.142663 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.144726 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.145942 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.147840 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.148307 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.148407 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.153126 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.153134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.153139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.153178 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.155782 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.156537 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.156545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.156549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.156690 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.157162 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.157168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.157173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.157473 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.157584 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.158305 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.158486 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260926 11:35:48.160574 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.140573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.98.1:60101)
0926 11:35:48.141086 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.141148 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.142146 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.142679 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.142688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.143386 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:48.144842 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.144849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.145855 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.145858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.147698 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.147710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.148497 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.148501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.148854 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.149731 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:48.149752 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:48.153126 (+ 3374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:48.155728 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.155735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.155756 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.156290 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.156889 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.156891 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.156893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.156995 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.157522 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:48.157529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:48.157949 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.158835 (+ 886us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.159530 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":685,"spinlock_wait_cycles":56832,"thread_start_us":166,"threads_started":1}
I20260926 11:35:48.160576 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.140573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.98.1:39911)
0926 11:35:48.141086 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.141140 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.142141 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.142682 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.142689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.143806 (+ 1117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:48.144550 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.144565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.146096 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.146102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.147667 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.147690 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.148605 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.148611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.148805 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.149697 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:48.149731 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:48.152825 (+ 3094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:48.155934 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.155941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.155947 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.156387 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.156830 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.156836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.156840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.157021 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.157657 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:48.157663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:48.158014 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.159010 (+ 996us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.159439 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":351,"spinlock_wait_cycles":22400,"thread_start_us":116,"threads_started":1}
I20260926 11:35:48.160787 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.141652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:39097)
0926 11:35:48.141903 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.141919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.142141 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.142875 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.142878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.143389 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:48.144489 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.144510 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.145696 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.145699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.147698 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.147710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.148497 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.148501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.148832 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.149857 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:48.149873 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:48.153211 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:48.155899 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.155902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.155904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.156290 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.156799 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.156802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.156804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.156997 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.157522 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:48.157529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:48.157949 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.158835 (+ 886us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.159309 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":471,"spinlock_wait_cycles":1664,"thread_start_us":95,"threads_started":1}
I20260926 11:35:48.162178 7474 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41515
I20260926 11:35:48.162271 7473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43973
I20260926 11:35:48.162439 7474 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:48.162441 7473 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:48.162906 7475 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41513
I20260926 11:35:48.163012 7475 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:48.163143 7473 heartbeater.cc:511] Master 127.0.98.62:43973 requested a full tablet report, sending...
I20260926 11:35:48.163184 7475 heartbeater.cc:511] Master 127.0.98.61:41513 requested a full tablet report, sending...
I20260926 11:35:48.163718 7474 heartbeater.cc:511] Master 127.0.98.60:41515 requested a full tablet report, sending...
I20260926 11:35:48.164969 7111 ts_manager.cc:194] Registered new tserver with Master: 4352e835c189494db1abdfadc82db152 (127.0.98.1:34257)
I20260926 11:35:48.164976 7297 ts_manager.cc:194] Registered new tserver with Master: 4352e835c189494db1abdfadc82db152 (127.0.98.1:34257)
I20260926 11:35:48.165699 6509 ts_manager.cc:194] Re-registered known tserver with Master: 4352e835c189494db1abdfadc82db152 (127.0.98.1:34257)
I20260926 11:35:48.166337 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39097
I20260926 11:35:48.228322 6968 heartbeater.cc:503] Master 127.0.98.62:43973 was elected leader, sending a full tablet report...
W20260926 11:35:48.282830 7481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:48.283228 7481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:48.283329 7481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:48.288551 7481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:48.288749 7481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:48.288915 7481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:48.295156 7481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46201
--tserver_master_addrs=127.0.98.62:43973,127.0.98.61:41513,127.0.98.60:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:48.296446 7481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:48.297586 7481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:48.309103 7487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:48.309175 7481 server_base.cc:1034] running on GCE node
W20260926 11:35:48.309121 7486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:48.309165 7489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:48.309896 7481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:48.310717 7481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:48.311941 7481 hybrid_clock.cc:648] HybridClock initialized: now 1790422548311923 us; error 39 us; skew 500 ppm
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:48.315495 7481 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 11:35:48.316610 7481 webserver.cc:460] Webserver started at http://127.0.98.2:46201/ using document root <none> and password file <none>
I20260926 11:35:48.317284 7481 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:48.317417 7481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:48.322295 7481 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260926 11:35:48.325083 7496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.326694 7481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:48.326880 7481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "6a4b1009d5894659b17fc8e09f3f7e98"
format_stamp: "Formatted at 2026-09-26 11:35:46 on dist-test-slave-4pw5"
server_key: "d93c093fdcde6cb3600d704491e0d2a1"
server_key_iv: "159a349ec67e990146653bfe8b96c7b0"
server_key_version: "encryptionkey@0"
I20260926 11:35:48.327513 7481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:48.341673 7481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:48.345698 7481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:48.345883 7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:48.346474 7481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:48.347410 7481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:48.347463 7481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.347512 7481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:48.347540 7481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.358511 7481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39887
I20260926 11:35:48.358565 7609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39887 every 8 connection(s)
I20260926 11:35:48.359797 7481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:48.370075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 7481
I20260926 11:35:48.370275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 6839
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:48.381484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45343
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43973
--webserver_port=36941
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43973,127.0.98.61:41513
--tserver_master_addrs=127.0.98.62:43973,127.0.98.61:41513,127.0.98.60:41515 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 11:35:48.386219 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.361988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40721 (local address 127.0.98.61:41513)
0926 11:35:48.362129 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.362134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.363166 (+ 1032us) server_negotiation.cc:407] Connection header received
0926 11:35:48.364579 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.364583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.364642 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.364726 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.366751 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.367620 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.369393 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.369723 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.369813 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.379690 (+ 9877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.379694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.379698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.379732 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.382001 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.383911 (+ 1910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.383917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.383920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.384031 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.384570 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.384575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.384577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.384830 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.384934 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.385802 (+ 868us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.385955 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:35:48.387431 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.363659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:57805)
0926 11:35:48.363938 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.363955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.364321 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.365017 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.365020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.365630 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:48.366564 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.366588 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.368498 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.368503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.369220 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.369230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.369987 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.369991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.370426 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.371321 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:48.371354 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:48.379403 (+ 8049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:48.383088 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.383097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.383120 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.383771 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.384214 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.384220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.384222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.384454 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.385052 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:48.385058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:48.385485 (+ 427us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.385720 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.386129 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":249,"spinlock_wait_cycles":36352,"thread_start_us":101,"threads_started":1}
I20260926 11:35:48.387851 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.362487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.98.2:40721)
0926 11:35:48.362998 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.363055 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.364321 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.364906 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.364913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.365630 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:48.366564 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.366588 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.367971 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.367974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.369242 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.369253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.369987 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.369991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.370817 (+ 826us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.371351 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:48.371366 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:48.379403 (+ 8037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:48.383088 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.383097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.383120 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.383771 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.384192 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.384197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.384200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.384449 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.385006 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:48.385015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:48.385486 (+ 471us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.386395 (+ 909us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.386575 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":229,"spinlock_wait_cycles":59520,"thread_start_us":169,"threads_started":1}
I20260926 11:35:48.388872 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.363813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57805 (local address 127.0.98.62:43973)
0926 11:35:48.363947 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.363951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.364053 (+ 102us) server_negotiation.cc:407] Connection header received
0926 11:35:48.364851 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.364855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.364899 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.364974 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.367723 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.368349 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.369352 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.369694 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.369801 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.379690 (+ 9889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.379694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.379697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.379732 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.382915 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.383918 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.383924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.383930 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.384051 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.384611 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.384615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.384617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.384891 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.384996 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.388530 (+ 3534us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.388685 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:35:48.389019 7612 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41513
I20260926 11:35:48.389247 7612 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:48.389598 7610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43973
I20260926 11:35:48.389689 7610 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:48.389880 7610 heartbeater.cc:511] Master 127.0.98.62:43973 requested a full tablet report, sending...
I20260926 11:35:48.389880 7612 heartbeater.cc:511] Master 127.0.98.61:41513 requested a full tablet report, sending...
I20260926 11:35:48.391163 6509 ts_manager.cc:194] Re-registered known tserver with Master: 6a4b1009d5894659b17fc8e09f3f7e98 (127.0.98.2:39887)
I20260926 11:35:48.391235 7111 ts_manager.cc:194] Registered new tserver with Master: 6a4b1009d5894659b17fc8e09f3f7e98 (127.0.98.2:39887)
I20260926 11:35:48.391433 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.362095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43947 (local address 127.0.98.60:41515)
0926 11:35:48.362232 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.362236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.363167 (+ 931us) server_negotiation.cc:407] Connection header received
0926 11:35:48.369471 (+ 6304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.369475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.369541 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.369592 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.377752 (+ 8160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.378644 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.382567 (+ 3923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.382990 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.383086 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.386274 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.386279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.386282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.386317 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.389165 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.389865 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.389872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.389877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.390035 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.390363 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.390370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.390375 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.390632 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.390739 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.391116 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.391271 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:35:48.392084 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57805
I20260926 11:35:48.392272 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.362488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.98.2:43947)
0926 11:35:48.362997 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.363055 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.364318 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.369724 (+ 5406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.369726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.376535 (+ 6809us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:48.376720 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.376733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.381014 (+ 4281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.381018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.382385 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.382400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.383319 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.383323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.383404 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.384274 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:48.384291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:48.386078 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:48.389434 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.389438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.389440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.389738 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.390160 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.390162 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.390164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.390243 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.390787 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:48.390791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:48.390896 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.391837 (+ 941us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.392051 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"spinlock_wait_cycles":11776,"thread_start_us":143,"threads_started":1}
I20260926 11:35:48.393340 7611 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41515
I20260926 11:35:48.393446 7611 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:48.393828 7611 heartbeater.cc:511] Master 127.0.98.60:41515 requested a full tablet report, sending...
I20260926 11:35:48.394862 7297 ts_manager.cc:194] Registered new tserver with Master: 6a4b1009d5894659b17fc8e09f3f7e98 (127.0.98.2:39887)
I20260926 11:35:48.417969 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.414142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.0.98.60:41515)
0926 11:35:48.414286 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.414289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.414306 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:35:48.414428 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.414430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.414483 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.414566 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:48.414971 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.415482 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.416902 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.417387 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.417488 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.417623 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.417778 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.417853 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:35:48.417964 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.413988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.0.1:50430)
0926 11:35:48.414182 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.414200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.414314 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.414694 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.414699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.414724 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:48.414838 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.414845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.415618 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.415621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.416746 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.416758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.417522 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.417526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.417619 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.417645 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.417730 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.417811 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 11:35:48.418776 7307 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 11:35:48.419194 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de9bb59f0b048119f50c9f51ddca9a2. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:48.419351 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: This master's current role is: LEARNER
W20260926 11:35:48.511324 7617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:48.511698 7617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:48.511783 7617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:48.516083 7617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:48.516173 7617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:48.516263 7617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:48.521951 7617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36941
--tserver_master_addrs=127.0.98.62:43973,127.0.98.61:41513,127.0.98.60:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:48.523429 7617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:48.524864 7617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:48.536388 7623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:48.536568 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:48.536485 7622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:48.536758 7617 server_base.cc:1034] running on GCE node
I20260926 11:35:48.537382 7617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:48.538182 7617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:48.539355 7617 hybrid_clock.cc:648] HybridClock initialized: now 1790422548539321 us; error 49 us; skew 500 ppm
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:48.542541 7617 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 11:35:48.543793 7617 webserver.cc:460] Webserver started at http://127.0.98.3:36941/ using document root <none> and password file <none>
I20260926 11:35:48.544430 7617 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:48.544509 7617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:48.548792 7617 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:35:48.551146 7632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.552681 7617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:48.552839 7617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "f8526fc109694c32a64b01e2ce5c2331"
format_stamp: "Formatted at 2026-09-26 11:35:47 on dist-test-slave-4pw5"
server_key: "ad44e1ed3fc8d28268b4b7c1c23016e4"
server_key_iv: "f7e6ad3ea1f7b80e728260c185b2989d"
server_key_version: "encryptionkey@0"
I20260926 11:35:48.553454 7617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:48.573089 7617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:48.576907 7617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:48.577152 7617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:48.577783 7617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:48.578836 7617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:48.578915 7617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.578987 7617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:48.579039 7617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:48.588407 7617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45343
I20260926 11:35:48.588459 7745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45343 every 8 connection(s)
I20260926 11:35:48.589504 7617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:48.590758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 7617
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 11:35:48.606025 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.593702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.0.98.60:41515)
0926 11:35:48.593828 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.593832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.595358 (+ 1526us) server_negotiation.cc:407] Connection header received
0926 11:35:48.595456 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.595460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.595520 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.595653 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.596559 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.597293 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.598430 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.598743 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.598826 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.601466 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.601470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.601486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.601514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.603343 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.604383 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.604386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.604389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.604505 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.604892 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.604898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.604902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.605164 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.605272 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.605713 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.605845 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":54}
I20260926 11:35:48.606904 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.593147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.0.98.62:43973)
0926 11:35:48.593290 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.593294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.594468 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 11:35:48.594704 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.594708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.594761 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.594858 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.596026 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.596852 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.598012 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.598357 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.598436 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.600502 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.600507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.600510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.600543 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.603348 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.604034 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.604040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.604043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.604140 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.605713 (+ 1573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.605716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.605719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.605939 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.606068 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.606629 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.606761 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:35:48.609820 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.593056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.0.98.61:41513)
0926 11:35:48.593627 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.593631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.595072 (+ 1441us) server_negotiation.cc:407] Connection header received
0926 11:35:48.595794 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.595796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.595827 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.595882 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.596862 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.597621 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.599298 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.599717 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.599795 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.601915 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.601920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.601924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.601959 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.604087 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.604615 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.604619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.604620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.604711 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.608568 (+ 3857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.608572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.608574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.609021 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.609164 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.609491 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.609629 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":74,"threads_started":1}
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 11:35:48.612180 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.591788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33417 (local address 127.0.98.61:41513)
0926 11:35:48.591930 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.591936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.594294 (+ 2358us) server_negotiation.cc:407] Connection header received
0926 11:35:48.595678 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.595681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.595738 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.595794 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.597733 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.598485 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.602382 (+ 3897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.602719 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.602818 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.606437 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.606444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.606450 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.606477 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.608842 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.609556 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.609559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.609561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.609644 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.610310 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.610316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.610321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.610674 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.610768 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.611863 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.611996 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 11:35:48 dist-test-slave-4pw5 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:48.613875 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.592385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.98.3:33417)
0926 11:35:48.594148 (+ 1763us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.594209 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.595401 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.596246 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.596259 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.596795 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:48.597482 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.597499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.598664 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.598669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.602229 (+ 3560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.602245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.603055 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.603061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.603318 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.604028 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:48.604051 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:48.605990 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:48.608999 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.609010 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.609033 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.609409 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.609746 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.609749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.609751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.610195 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.610841 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:48.610851 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:48.611491 (+ 640us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.612572 (+ 1081us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.612934 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"mutex_wait_us":27,"thread_start_us":791,"threads_started":1}
I20260926 11:35:48.610726 6509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 11:35:48.615240 7748 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41513
W20260926 11:35:48.615365 6509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:48.615509 7748 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:48.616201 7748 heartbeater.cc:511] Master 127.0.98.61:41513 requested a full tablet report, sending...
I20260926 11:35:48.617511 7111 ts_manager.cc:194] Registered new tserver with Master: f8526fc109694c32a64b01e2ce5c2331 (127.0.98.3:45343)
I20260926 11:35:48.618789 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.594337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41715 (local address 127.0.98.60:41515)
0926 11:35:48.595622 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.595628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.598996 (+ 3368us) server_negotiation.cc:407] Connection header received
0926 11:35:48.599041 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.599044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.599100 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.599189 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.600044 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.600825 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.604913 (+ 4088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.605291 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.605380 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.612170 (+ 6790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.612174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.612177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.612207 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.616182 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.617252 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.617256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.617258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.617365 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.617697 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.617700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.617702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.617962 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.618053 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.618456 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.618622 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":1164,"thread_start_us":63,"threads_started":1}
I20260926 11:35:48.618868 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: true } } }
I20260926 11:35:48.618999 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:48.619398 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.594222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.98.3:41715)
0926 11:35:48.598849 (+ 4627us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.598867 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.598969 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.599427 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.599431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.599706 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:48.599871 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.599880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.600994 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.600998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.603770 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.603815 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.607722 (+ 3907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.607728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.607893 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.608410 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:48.608426 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:48.611244 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:48.616405 (+ 5161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.616409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.616411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.616722 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.617475 (+ 753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.617480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.617482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.617577 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.618113 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:48.618115 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:48.618241 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.618983 (+ 742us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.619170 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4550,"mutex_wait_us":42,"thread_start_us":108,"threads_started":1}
I20260926 11:35:48.619848 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.593618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:40285)
0926 11:35:48.594782 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.594795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.597606 (+ 2811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.604335 (+ 6729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.604339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.604540 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:48.604704 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.604711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.607714 (+ 3003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.607719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.609753 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.609771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.610628 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.610632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.610746 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.611396 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:48.611410 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:48.614515 (+ 3105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:48.617950 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.617953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.617955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.618215 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.618767 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:48.618769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:48.618771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:48.618847 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:48.619293 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:48.619296 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:48.619395 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.619504 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.619683 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"thread_start_us":117,"threads_started":1}
I20260926 11:35:48.620003 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41515
I20260926 11:35:48.620016 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.594742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40285 (local address 127.0.98.62:43973)
0926 11:35:48.602408 (+ 7666us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.602413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.602428 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:35:48.602462 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.602465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.602525 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.602634 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.605928 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.607154 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.609900 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.610281 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.610381 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.614765 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.614770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.614774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.614805 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.617786 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.618312 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.618315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.618317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.618445 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.618959 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.618962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.618964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.619202 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.619302 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.619731 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.619855 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7572,"thread_start_us":79,"threads_started":1}
I20260926 11:35:48.620083 7747 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:48.620280 7747 heartbeater.cc:511] Master 127.0.98.60:41515 requested a full tablet report, sending...
I20260926 11:35:48.620805 7746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43973
I20260926 11:35:48.620882 7746 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:48.621006 7746 heartbeater.cc:511] Master 127.0.98.62:43973 requested a full tablet report, sending...
I20260926 11:35:48.621610 6508 ts_manager.cc:194] Re-registered known tserver with Master: f8526fc109694c32a64b01e2ce5c2331 (127.0.98.3:45343)
I20260926 11:35:48.622375 6508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40285
I20260926 11:35:48.622576 7297 ts_manager.cc:194] Registered new tserver with Master: f8526fc109694c32a64b01e2ce5c2331 (127.0.98.3:45343)
I20260926 11:35:48.623521 7069 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: attempting to promote NON_VOTER 3865d02fa61c4cfea5debb15f877ed27 to VOTER
I20260926 11:35:48.623886 7069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } }
I20260926 11:35:48.625634 7307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 LEARNER]: Refusing update from remote peer 3de9bb59f0b048119f50c9f51ddca9a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 11:35:48.625725 7120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Refusing update from remote peer 3de9bb59f0b048119f50c9f51ddca9a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 11:35:48.626021 7768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 11:35:48.626206 7764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 11:35:48.628132 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3865d02fa61c4cfea5debb15f877ed27 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:48.628614 7120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3865d02fa61c4cfea5debb15f877ed27 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:48.628602 7307 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3865d02fa61c4cfea5debb15f877ed27 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:48.629886 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:48.630019 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:48.630141 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:48.630215 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:48.630888 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:48.630873 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:48.631007 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:35:48.631019 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:35:48.632373 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:48.632475 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:48.643738 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.637440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34257 (local address 127.0.0.1:53736)
0926 11:35:48.637626 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.637642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.637726 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.639302 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.639305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.639321 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:48.639430 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.639439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.640800 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.640806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.642089 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.642103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.643251 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.643255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.643344 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.643366 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.643426 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.643512 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 11:35:48.644321 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.637600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.0.98.1:34257)
0926 11:35:48.638801 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.638818 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.638842 (+ 24us) server_negotiation.cc:407] Connection header received
0926 11:35:48.638930 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.638935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.639164 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.639300 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:48.639561 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.640685 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.642251 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.643110 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.643242 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.643424 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.643564 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.644206 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094,"thread_start_us":95,"threads_started":1}
I20260926 11:35:48.644676 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.638345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45343 (local address 127.0.0.1:37404)
0926 11:35:48.639541 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.639555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.639681 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.640187 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.640190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.640208 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:48.640293 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.640298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.641557 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.641561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.642699 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.642708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.644073 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.644076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.644286 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.644345 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.644501 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.644557 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"mutex_wait_us":75,"thread_start_us":133,"threads_started":1}
I20260926 11:35:48.645022 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.637831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.0.98.2:39887)
0926 11:35:48.638183 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.638228 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.638757 (+ 529us) server_negotiation.cc:407] Connection header received
0926 11:35:48.638961 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.638969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.639164 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.639331 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:48.639823 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.641203 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.642725 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.643689 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.643779 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.643924 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.644752 (+ 828us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.644873 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":134,"threads_started":1}
I20260926 11:35:48.645210 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.638518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.0.98.3:45343)
0926 11:35:48.638836 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.638848 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.639650 (+ 802us) server_negotiation.cc:407] Connection header received
0926 11:35:48.639796 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.639803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.640026 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.640188 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:48.640387 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.641422 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.642830 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.643689 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.643790 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.644026 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.644867 (+ 841us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.644970 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":117,"threads_started":1}
I20260926 11:35:48.644677 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.637655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39887 (local address 127.0.0.1:51428)
0926 11:35:48.638636 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.638648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.638834 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.639549 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.639551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.639564 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:48.639697 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.639703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.641341 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.641344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.642578 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.642588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.644035 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.644041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.644283 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.644344 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.644422 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.644519 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":101,"thread_start_us":109,"threads_started":1}
I20260926 11:35:48.648272 7544 tablet_service.cc:1467] Processing CreateTablet for tablet 78b4456da32b43a28ad8679cf44e42d5 (DEFAULT_TABLE table=test-table [id=abaf07cdaeaa44d1adb75cb30ba5eb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:48.648273 7680 tablet_service.cc:1467] Processing CreateTablet for tablet 78b4456da32b43a28ad8679cf44e42d5 (DEFAULT_TABLE table=test-table [id=abaf07cdaeaa44d1adb75cb30ba5eb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:48.648273 7407 tablet_service.cc:1467] Processing CreateTablet for tablet 78b4456da32b43a28ad8679cf44e42d5 (DEFAULT_TABLE table=test-table [id=abaf07cdaeaa44d1adb75cb30ba5eb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:48.649374 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b4456da32b43a28ad8679cf44e42d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:48.649380 7407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b4456da32b43a28ad8679cf44e42d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:48.649380 7680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b4456da32b43a28ad8679cf44e42d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:48.659287 7780 tablet_bootstrap.cc:492] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: Bootstrap starting.
I20260926 11:35:48.660293 7779 tablet_bootstrap.cc:492] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: Bootstrap starting.
I20260926 11:35:48.661937 7780 tablet_bootstrap.cc:654] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:48.662446 7779 tablet_bootstrap.cc:654] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:48.662778 7780 log.cc:824] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:48.663196 7779 log.cc:824] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:48.663985 7781 tablet_bootstrap.cc:492] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: Bootstrap starting.
I20260926 11:35:48.665114 7780 tablet_bootstrap.cc:492] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: No bootstrap required, opened a new log
I20260926 11:35:48.665212 7779 tablet_bootstrap.cc:492] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: No bootstrap required, opened a new log
I20260926 11:35:48.665310 7780 ts_tablet_manager.cc:1397] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:35:48.665375 7779 ts_tablet_manager.cc:1397] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:35:48.666303 7781 tablet_bootstrap.cc:654] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:48.667152 7781 log.cc:824] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:48.669848 7781 tablet_bootstrap.cc:492] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: No bootstrap required, opened a new log
I20260926 11:35:48.670061 7781 ts_tablet_manager.cc:1397] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:35:48.670095 7779 raft_consensus.cc:348] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } }
I20260926 11:35:48.670356 7779 raft_consensus.cc:374] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:48.670395 7779 raft_consensus.cc:729] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4b1009d5894659b17fc8e09f3f7e98, State: Initialized, Role: FOLLOWER
I20260926 11:35:48.670357 7780 raft_consensus.cc:348] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } }
I20260926 11:35:48.670599 7780 raft_consensus.cc:374] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:48.670657 7780 raft_consensus.cc:729] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4352e835c189494db1abdfadc82db152, State: Initialized, Role: FOLLOWER
I20260926 11:35:48.671119 7780 consensus_queue.cc:260] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } }
I20260926 11:35:48.672056 7473 heartbeater.cc:503] Master 127.0.98.62:43973 was elected leader, sending a full tablet report...
I20260926 11:35:48.672133 7779 consensus_queue.cc:260] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } }
I20260926 11:35:48.672500 7780 ts_tablet_manager.cc:1428] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 11:35:48.673274 7779 ts_tablet_manager.cc:1428] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 11:35:48.673744 7610 heartbeater.cc:503] Master 127.0.98.62:43973 was elected leader, sending a full tablet report...
I20260926 11:35:48.675941 7781 raft_consensus.cc:348] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } }
I20260926 11:35:48.676337 7781 raft_consensus.cc:374] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:48.676421 7781 raft_consensus.cc:729] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8526fc109694c32a64b01e2ce5c2331, State: Initialized, Role: FOLLOWER
I20260926 11:35:48.676927 7781 consensus_queue.cc:260] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } }
I20260926 11:35:48.677660 7746 heartbeater.cc:503] Master 127.0.98.62:43973 was elected leader, sending a full tablet report...
I20260926 11:35:48.678064 7781 ts_tablet_manager.cc:1428] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 11:35:48.714411 7786 raft_consensus.cc:482] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:48.714738 7786 raft_consensus.cc:504] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } }
I20260926 11:35:48.716148 7786 leader_election.cc:288] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4352e835c189494db1abdfadc82db152 (127.0.98.1:34257), f8526fc109694c32a64b01e2ce5c2331 (127.0.98.3:45343)
I20260926 11:35:48.720961 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.716807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45343 (local address 127.0.98.2:44115)
0926 11:35:48.716937 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.716953 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.717088 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.717361 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.717364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.717381 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:48.717497 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.717504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.718428 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.718431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.719551 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.719562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.720483 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.720486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.720580 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.720605 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.720676 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.720771 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":64}
I20260926 11:35:48.721011 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.716995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44115 (local address 127.0.98.3:45343)
0926 11:35:48.717136 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.717140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.717153 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:35:48.717209 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.717212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.717258 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.717331 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:48.717663 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.718297 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.719736 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.720354 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.720452 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.720513 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.720802 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.720881 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:35:48.721011 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.716635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51481 (local address 127.0.98.1:34257)
0926 11:35:48.716782 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.716786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.716800 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:35:48.716877 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.716880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.716934 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.717042 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:48.717363 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.718051 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.719455 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.720176 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.720308 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.720384 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.720802 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.720881 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 11:35:48.720966 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.716463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34257 (local address 127.0.98.2:51481)
0926 11:35:48.716596 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:48.716613 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:48.716721 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:48.717069 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:48.717073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:48.717097 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:48.717235 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.717241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.718221 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.718225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.719325 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:48.719336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.720350 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.720355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:48.720496 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.720526 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 11:35:48.720781 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 11:35:48.720838 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":89}
I20260926 11:35:48.721771 7427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b4456da32b43a28ad8679cf44e42d5" candidate_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4352e835c189494db1abdfadc82db152" is_pre_election: true
I20260926 11:35:48.721901 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b4456da32b43a28ad8679cf44e42d5" candidate_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8526fc109694c32a64b01e2ce5c2331" is_pre_election: true
I20260926 11:35:48.722152 7427 raft_consensus.cc:2454] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4b1009d5894659b17fc8e09f3f7e98 in term 0.
I20260926 11:35:48.722208 7700 raft_consensus.cc:2454] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4b1009d5894659b17fc8e09f3f7e98 in term 0.
I20260926 11:35:48.722715 7500 leader_election.cc:302] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a4b1009d5894659b17fc8e09f3f7e98, f8526fc109694c32a64b01e2ce5c2331; no voters:
I20260926 11:35:48.723021 7786 raft_consensus.cc:2781] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:48.723102 7786 raft_consensus.cc:482] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:48.723152 7786 raft_consensus.cc:3037] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:48.724633 7786 raft_consensus.cc:504] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } }
I20260926 11:35:48.725041 7786 leader_election.cc:288] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [CANDIDATE]: Term 1 election: Requested vote from peers 4352e835c189494db1abdfadc82db152 (127.0.98.1:34257), f8526fc109694c32a64b01e2ce5c2331 (127.0.98.3:45343)
I20260926 11:35:48.725528 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b4456da32b43a28ad8679cf44e42d5" candidate_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8526fc109694c32a64b01e2ce5c2331"
I20260926 11:35:48.725580 7427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b4456da32b43a28ad8679cf44e42d5" candidate_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4352e835c189494db1abdfadc82db152"
I20260926 11:35:48.725673 7700 raft_consensus.cc:3037] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:48.725764 7427 raft_consensus.cc:3037] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:48.726970 7427 raft_consensus.cc:2454] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4b1009d5894659b17fc8e09f3f7e98 in term 1.
I20260926 11:35:48.727012 7700 raft_consensus.cc:2454] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a4b1009d5894659b17fc8e09f3f7e98 in term 1.
I20260926 11:35:48.727465 7500 leader_election.cc:302] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a4b1009d5894659b17fc8e09f3f7e98, f8526fc109694c32a64b01e2ce5c2331; no voters:
I20260926 11:35:48.727826 7786 raft_consensus.cc:2781] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:48.728099 7786 raft_consensus.cc:686] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 1 LEADER]: Becoming Leader. State: Replica: 6a4b1009d5894659b17fc8e09f3f7e98, State: Running, Role: LEADER
I20260926 11:35:48.728600 7786 consensus_queue.cc:237] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } }
I20260926 11:35:48.732463 6509 catalog_manager.cc:5495] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a4b1009d5894659b17fc8e09f3f7e98 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:48.752776 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.749051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.0.98.2:39887)
0926 11:35:48.749197 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.749203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.749217 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:35:48.749338 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.749341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.749409 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.749526 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:48.749806 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.750326 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.751174 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.751419 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.751541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.751899 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:48.752354 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:48.752493 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.752550 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.752614 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:35:48.759186 7427 raft_consensus.cc:1261] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [term 1 FOLLOWER]: Refusing update from remote peer 6a4b1009d5894659b17fc8e09f3f7e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:48.759184 7700 raft_consensus.cc:1261] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [term 1 FOLLOWER]: Refusing update from remote peer 6a4b1009d5894659b17fc8e09f3f7e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:48.759999 7786 consensus_queue.cc:1035] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:48.760216 7788 consensus_queue.cc:1035] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:48.782372 6509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:48.782600 6509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:48.787202 6509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3de9bb59f0b048119f50c9f51ddca9a2: Sending DeleteTablet for 3 replicas of tablet 78b4456da32b43a28ad8679cf44e42d5
I20260926 11:35:48.788066 7407 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b4456da32b43a28ad8679cf44e42d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53736
I20260926 11:35:48.788461 7680 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b4456da32b43a28ad8679cf44e42d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37404
I20260926 11:35:48.788708 7544 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b4456da32b43a28ad8679cf44e42d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51428
I20260926 11:35:48.788775 7802 tablet_replica.cc:330] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: stopping tablet replica
I20260926 11:35:48.789212 7803 tablet_replica.cc:330] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: stopping tablet replica
I20260926 11:35:48.789383 7802 raft_consensus.cc:2229] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:48.789485 7803 raft_consensus.cc:2229] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:48.789839 7803 raft_consensus.cc:2258] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:48.789839 7802 raft_consensus.cc:2258] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:48.790251 7804 tablet_replica.cc:330] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: stopping tablet replica
I20260926 11:35:48.790591 7804 raft_consensus.cc:2229] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:48.791028 7804 raft_consensus.cc:2258] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:48.792486 7802 ts_tablet_manager.cc:1905] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:48.793098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 6477
I20260926 11:35:48.794334 7803 ts_tablet_manager.cc:1905] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:48.792799 7804 ts_tablet_manager.cc:1905] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:48.796442 7802 ts_tablet_manager.cc:1918] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:48.796540 7802 log.cc:1196] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/78b4456da32b43a28ad8679cf44e42d5
I20260926 11:35:48.796931 7802 ts_tablet_manager.cc:1939] T 78b4456da32b43a28ad8679cf44e42d5 P 4352e835c189494db1abdfadc82db152: Deleting consensus metadata
I20260926 11:35:48.798337 7804 ts_tablet_manager.cc:1918] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:48.798444 7804 log.cc:1196] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/78b4456da32b43a28ad8679cf44e42d5
I20260926 11:35:48.798751 7804 ts_tablet_manager.cc:1939] T 78b4456da32b43a28ad8679cf44e42d5 P 6a4b1009d5894659b17fc8e09f3f7e98: Deleting consensus metadata
W20260926 11:35:48.800887 7500 connection.cc:535] server connection from 127.0.0.1:51428 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 11:35:48.801738 7360 connection.cc:535] server connection from 127.0.0.1:53736 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 11:35:48.801960 7803 ts_tablet_manager.cc:1918] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:48.802058 7803 log.cc:1196] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/78b4456da32b43a28ad8679cf44e42d5
I20260926 11:35:48.802654 7803 ts_tablet_manager.cc:1939] T 78b4456da32b43a28ad8679cf44e42d5 P f8526fc109694c32a64b01e2ce5c2331: Deleting consensus metadata
W20260926 11:35:48.803963 7634 connection.cc:441] server connection from 127.0.0.1:37404 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37404 (request call id 1) could send its response
I20260926 11:35:48.812491 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.791829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.0.98.61:41513)
0926 11:35:48.791935 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.791941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.801366 (+ 9425us) server_negotiation.cc:407] Connection header received
0926 11:35:48.802184 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.802190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.802250 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.802308 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.803515 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.804298 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.805728 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.806048 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.806139 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.808177 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.808181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.808184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.808212 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.810344 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.810836 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.810842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.810847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.810938 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.811242 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.811246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.811248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.811539 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.811772 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.812153 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.812283 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 11:35:48.815785 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.804288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.0.98.61:41513)
0926 11:35:48.804467 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.804473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.805190 (+ 717us) server_negotiation.cc:407] Connection header received
0926 11:35:48.805358 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.805361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.805410 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.805458 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.806404 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.807326 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.808542 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.808923 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.808994 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.810719 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.810723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.810726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.810751 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.812741 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.814422 (+ 1681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.814425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.814427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.814508 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.814812 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.814815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.814817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.815010 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.815107 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.815435 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.815621 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 11:35:48.816855 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.805704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.0.98.60:41515)
0926 11:35:48.805827 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.805831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.806668 (+ 837us) server_negotiation.cc:407] Connection header received
0926 11:35:48.806856 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.806860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.806918 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.806981 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.808182 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.809007 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.810300 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.810654 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.810737 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.812461 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.812464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.812467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.812493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.814802 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.815293 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.815296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.815298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.815384 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.815706 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.815709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.815711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.815905 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.815996 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.816580 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.816701 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:35:48.822589 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:48.791718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.98.60:41515)
0926 11:35:48.791867 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:48.791874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:48.813066 (+ 21192us) server_negotiation.cc:407] Connection header received
0926 11:35:48.813221 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:48.813224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:48.813270 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:48.813333 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:48.814235 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.815109 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.816294 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:48.816634 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:48.816722 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:48.818401 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:48.818405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:48.818407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:48.818432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:48.820405 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.820875 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.820881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.820883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.820987 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:48.821557 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:48.821561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:48.821563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:48.821795 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:48.821887 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:48.822200 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 11:35:48.822330 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 11:35:49.772938 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:49.772278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:50661)
0926 11:35:49.772623 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:49.772779 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
W20260926 11:35:49.773574 7746 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:49.774312 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:49.773132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:43477)
0926 11:35:49.774065 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:49.774187 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":67,"threads_started":1}
W20260926 11:35:49.774755 7610 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:49.775316 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:49.774327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:40725)
0926 11:35:49.775127 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:49.775233 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":123,"threads_started":1}
W20260926 11:35:49.775751 7473 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:50.287470 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3de9bb59f0b048119f50c9f51ddca9a2)
I20260926 11:35:50.287817 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } }
I20260926 11:35:50.288077 7817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3de9bb59f0b048119f50c9f51ddca9a2)
I20260926 11:35:50.288271 7817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } }
I20260926 11:35:50.288986 7817 leader_election.cc:288] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973), f9ef07cd30064f14abcfb957e065e9fc (127.0.98.61:41513)
I20260926 11:35:50.289134 7816 leader_election.cc:288] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973), 3865d02fa61c4cfea5debb15f877ed27 (127.0.98.60:41515)
I20260926 11:35:50.289863 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.289307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47466)
0926 11:35:50.289657 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.289754 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":83,"threads_started":1}
I20260926 11:35:50.289867 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.289257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47458)
0926 11:35:50.289657 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.289754 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":111,"threads_started":1}
W20260926 11:35:50.290331 7280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
W20260926 11:35:50.290377 7097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:50.291477 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.291172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47474)
0926 11:35:50.291272 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.291353 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 11:35:50.291679 7280 leader_election.cc:334] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:50.292636 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.292135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47484)
0926 11:35:50.292442 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.292514 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":121,"threads_started":1}
W20260926 11:35:50.292925 7097 leader_election.cc:334] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:50.294750 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.289815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.0.98.60:41515)
0926 11:35:50.290019 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:50.290025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:50.290278 (+ 253us) server_negotiation.cc:407] Connection header received
0926 11:35:50.290477 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:50.290524 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:50.290578 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:50.290662 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:50.291224 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:50.292127 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:50.293584 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:50.294056 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:50.294229 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:50.294325 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:50.294430 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 11:35:50.294525 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260926 11:35:50.294751 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.290004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41515 (local address 127.0.0.1:50466)
0926 11:35:50.290145 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.290164 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:50.290337 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:50.290802 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:50.290808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:50.290822 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:50.291067 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:50.291075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:50.292244 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:50.292250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:50.293434 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:50.293446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:50.294188 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:50.294194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:50.294304 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:50.294330 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:35:50.294414 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 11:35:50.294519 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":38}
I20260926 11:35:50.294775 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.289306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41513 (local address 127.0.0.1:41956)
0926 11:35:50.289665 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.289687 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:50.289818 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:50.290853 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:50.290856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:50.290881 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:50.291035 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:50.291044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:50.292296 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:50.292302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:50.293416 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:50.293427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:50.294162 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:50.294166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:50.294310 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:50.294333 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:35:50.294420 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 11:35:50.294537 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":5,"thread_start_us":104,"threads_started":1}
I20260926 11:35:50.294763 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.290168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.0.98.61:41513)
0926 11:35:50.290574 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:50.290579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:50.290592 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:35:50.290649 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:50.290654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:50.290691 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:50.290817 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:50.291169 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:50.292208 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:50.293554 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:50.294023 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:50.294152 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:50.294221 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:50.294450 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 11:35:50.294547 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":222,"spinlock_wait_cycles":9728,"thread_start_us":72,"threads_started":1}
I20260926 11:35:50.295315 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3865d02fa61c4cfea5debb15f877ed27" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f9ef07cd30064f14abcfb957e065e9fc" is_pre_election: true
I20260926 11:35:50.295406 7306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9ef07cd30064f14abcfb957e065e9fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3865d02fa61c4cfea5debb15f877ed27" is_pre_election: true
I20260926 11:35:50.295498 7120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3865d02fa61c4cfea5debb15f877ed27 in term 1.
I20260926 11:35:50.295696 7306 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ef07cd30064f14abcfb957e065e9fc in term 1.
I20260926 11:35:50.295913 7278 leader_election.cc:302] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3865d02fa61c4cfea5debb15f877ed27, f9ef07cd30064f14abcfb957e065e9fc; no voters: 3de9bb59f0b048119f50c9f51ddca9a2
I20260926 11:35:50.296169 7817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 11:35:50.296169 7097 leader_election.cc:302] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3865d02fa61c4cfea5debb15f877ed27, f9ef07cd30064f14abcfb957e065e9fc; no voters: 3de9bb59f0b048119f50c9f51ddca9a2
I20260926 11:35:50.296227 7817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3de9bb59f0b048119f50c9f51ddca9a2)
I20260926 11:35:50.296284 7817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:35:50.296484 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 11:35:50.296551 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3de9bb59f0b048119f50c9f51ddca9a2)
I20260926 11:35:50.296592 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:35:50.297935 7817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } }
I20260926 11:35:50.297952 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } }
I20260926 11:35:50.298424 7816 leader_election.cc:288] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [CANDIDATE]: Term 2 election: Requested vote from peers 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973), 3865d02fa61c4cfea5debb15f877ed27 (127.0.98.60:41515)
I20260926 11:35:50.298725 7817 leader_election.cc:288] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 2 election: Requested vote from peers 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973), f9ef07cd30064f14abcfb957e065e9fc (127.0.98.61:41513)
I20260926 11:35:50.298914 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.298504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47498)
0926 11:35:50.298764 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.298828 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260926 11:35:50.299146 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.298894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47500)
0926 11:35:50.299013 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.299071 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 11:35:50.299391 7306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9ef07cd30064f14abcfb957e065e9fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3865d02fa61c4cfea5debb15f877ed27"
I20260926 11:35:50.299635 7306 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f9ef07cd30064f14abcfb957e065e9fc in current term 2: Already voted for candidate 3865d02fa61c4cfea5debb15f877ed27 in this term.
I20260926 11:35:50.299645 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.299409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47508)
0926 11:35:50.299511 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.299555 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 11:35:50.299793 7280 leader_election.cc:334] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:50.299882 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3865d02fa61c4cfea5debb15f877ed27" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f9ef07cd30064f14abcfb957e065e9fc"
I20260926 11:35:50.300062 7120 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3865d02fa61c4cfea5debb15f877ed27 in current term 2: Already voted for candidate f9ef07cd30064f14abcfb957e065e9fc in this term.
I20260926 11:35:50.300487 7278 leader_election.cc:302] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3865d02fa61c4cfea5debb15f877ed27; no voters: 3de9bb59f0b048119f50c9f51ddca9a2, f9ef07cd30064f14abcfb957e065e9fc
I20260926 11:35:50.300547 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.300298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47518)
0926 11:35:50.300453 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.300496 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 11:35:50.300740 7817 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260926 11:35:50.300907 7097 leader_election.cc:334] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:50.300983 7097 leader_election.cc:302] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9ef07cd30064f14abcfb957e065e9fc; no voters: 3865d02fa61c4cfea5debb15f877ed27, 3de9bb59f0b048119f50c9f51ddca9a2
I20260926 11:35:50.301120 7816 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 11:35:50.775000 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.774268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:45149)
0926 11:35:50.774796 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.774877 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":115,"threads_started":1}
I20260926 11:35:50.776369 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.775661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:43509)
0926 11:35:50.776122 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.776218 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260926 11:35:50.776722 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:50.776217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:38089)
0926 11:35:50.776530 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:50.776604 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":70,"threads_started":1}
I20260926 11:35:51.777055 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:51.776304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:49399)
0926 11:35:51.776834 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:51.776925 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":210,"threads_started":1}
I20260926 11:35:51.778335 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:51.777410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:56463)
0926 11:35:51.778086 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:51.778197 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":80,"threads_started":1}
I20260926 11:35:51.778594 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:51.777902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:55271)
0926 11:35:51.778386 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:51.778465 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":84,"threads_started":1}
I20260926 11:35:52.312901 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:52.313112 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } }
I20260926 11:35:52.313742 7832 leader_election.cc:288] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973), f9ef07cd30064f14abcfb957e065e9fc (127.0.98.61:41513)
I20260926 11:35:52.314292 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3865d02fa61c4cfea5debb15f877ed27" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f9ef07cd30064f14abcfb957e065e9fc" is_pre_election: true
I20260926 11:35:52.314364 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.313936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47782)
0926 11:35:52.314208 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.314267 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260926 11:35:52.314499 7120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3865d02fa61c4cfea5debb15f877ed27 in term 2.
I20260926 11:35:52.314837 7278 leader_election.cc:302] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3865d02fa61c4cfea5debb15f877ed27, f9ef07cd30064f14abcfb957e065e9fc; no voters:
I20260926 11:35:52.315068 7832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 11:35:52.315135 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:52.315179 7832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 2 FOLLOWER]: Advancing to term 3
I20260926 11:35:52.315419 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.315115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47794)
0926 11:35:52.315287 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.315359 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260926 11:35:52.315565 7280 leader_election.cc:334] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:52.316642 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } }
I20260926 11:35:52.317066 7832 leader_election.cc:288] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 3 election: Requested vote from peers 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973), f9ef07cd30064f14abcfb957e065e9fc (127.0.98.61:41513)
I20260926 11:35:52.317482 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.317204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47808)
0926 11:35:52.317344 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.317416 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 11:35:52.317698 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3865d02fa61c4cfea5debb15f877ed27" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f9ef07cd30064f14abcfb957e065e9fc"
I20260926 11:35:52.317899 7120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 2 FOLLOWER]: Advancing to term 3
I20260926 11:35:52.318475 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.318232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47824)
0926 11:35:52.318358 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.318414 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 11:35:52.318703 7280 leader_election.cc:334] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
I20260926 11:35:52.319339 7120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3865d02fa61c4cfea5debb15f877ed27 in term 3.
I20260926 11:35:52.319821 7278 leader_election.cc:302] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3865d02fa61c4cfea5debb15f877ed27, f9ef07cd30064f14abcfb957e065e9fc; no voters: 3de9bb59f0b048119f50c9f51ddca9a2
I20260926 11:35:52.320055 7832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 11:35:52.320338 7832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [term 3 LEADER]: Becoming Leader. State: Replica: 3865d02fa61c4cfea5debb15f877ed27, State: Running, Role: LEADER
I20260926 11:35:52.320686 7832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } }
I20260926 11:35:52.322151 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3865d02fa61c4cfea5debb15f877ed27. Latest consensus state: current_term: 3 leader_uuid: "3865d02fa61c4cfea5debb15f877ed27" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:52.322300 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:52.322664 7837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:52.325525 7837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:52.325996 7837 catalog_manager.cc:1259] Loaded cluster ID: 631d8c3567f841c3ac5ccae91a0a4419
I20260926 11:35:52.326046 7837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:52.326653 7837 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:52.327113 7837 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Loaded TSK: 0
I20260926 11:35:52.328092 7837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:35:52.404070 7296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:52.408413 7296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:52.412928 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.412512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47840)
0926 11:35:52.412738 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.412832 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 11:35:52.412930 7120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [term 3 FOLLOWER]: Refusing update from remote peer 3865d02fa61c4cfea5debb15f877ed27: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 11:35:52.413606 7832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 11:35:52.413887 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.413570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.0.1:47854)
0926 11:35:52.413720 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.413774 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 11:35:52.414009 7280 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 -> Peer 3de9bb59f0b048119f50c9f51ddca9a2 (127.0.98.62:43973): Couldn't send request to peer 3de9bb59f0b048119f50c9f51ddca9a2. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 11:35:52.415859 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3865d02fa61c4cfea5debb15f877ed27. Latest consensus state: current_term: 3 leader_uuid: "3865d02fa61c4cfea5debb15f877ed27" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:52.416054 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:35:52.417536 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3865d02fa61c4cfea5debb15f877ed27" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:52.417737 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:52.420814 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3865d02fa61c4cfea5debb15f877ed27" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3de9bb59f0b048119f50c9f51ddca9a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } peers { permanent_uuid: "f9ef07cd30064f14abcfb957e065e9fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41513 } attrs { promote: false } } peers { permanent_uuid: "3865d02fa61c4cfea5debb15f877ed27" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41515 } attrs { promote: false } } }
I20260926 11:35:52.421015 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ef07cd30064f14abcfb957e065e9fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:35:52.431419 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.425782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45343 (local address 127.0.0.1:37406)
0926 11:35:52.425952 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.425970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:52.426752 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:52.427212 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:52.427216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:52.427240 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:52.427406 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.427419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.428699 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.428703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.429947 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.429957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.430845 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.430849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.430964 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.430998 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 11:35:52.431088 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 11:35:52.431220 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":60,"spinlock_wait_cycles":7680}
I20260926 11:35:52.431406 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.425908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.0.98.3:45343)
0926 11:35:52.426444 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:52.426454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:52.426475 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:35:52.427005 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:52.427009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:52.427077 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:52.427509 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:52.427674 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.428544 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.430119 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.430702 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.430833 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.430928 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:52.431125 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 11:35:52.431221 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":135,"threads_started":1}
I20260926 11:35:52.431638 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.426151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39887 (local address 127.0.0.1:51436)
0926 11:35:52.426556 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.426575 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:52.426751 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:52.427335 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:52.427341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:52.427359 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:52.427501 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.427508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.428671 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.428677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.430211 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.430221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.431216 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.431219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.431293 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.431315 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:35:52.431374 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 11:35:52.431449 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":113,"spinlock_wait_cycles":100480,"thread_start_us":106,"threads_started":1}
I20260926 11:35:52.432189 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.426273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.0.98.2:39887)
0926 11:35:52.427045 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:52.427050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:52.427064 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:35:52.427129 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:52.427135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:52.427199 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:52.427294 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:52.427666 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.428545 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.430363 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.431024 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.431839 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.431920 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:52.431958 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 11:35:52.432046 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":77,"threads_started":1}
I20260926 11:35:52.432564 7680 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2 (DEFAULT_TABLE table=test-table [id=3b094e77e91340e797c9c474a843e276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:52.432984 7680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:52.432998 7544 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2 (DEFAULT_TABLE table=test-table [id=3b094e77e91340e797c9c474a843e276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:52.433122 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.426344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34257 (local address 127.0.0.1:53744)
0926 11:35:52.426656 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.426670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:52.426751 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:52.427490 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:52.427493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:52.427507 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:52.427633 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.427643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.429378 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.429384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.430520 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.430528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.432710 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.432714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.432810 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.432835 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 11:35:52.432922 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 11:35:52.432999 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":3,"spinlock_wait_cycles":2304,"thread_start_us":148,"threads_started":1}
I20260926 11:35:52.433230 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.426771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.0.98.1:34257)
0926 11:35:52.427109 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:52.427114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:52.427131 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:35:52.427193 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:52.427196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:52.427238 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:52.427353 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:52.427930 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.429159 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.430686 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.432560 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.432694 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.432768 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:52.432966 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 11:35:52.433067 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260926 11:35:52.433362 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:52.435716 7850 tablet_bootstrap.cc:492] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: Bootstrap starting.
I20260926 11:35:52.436412 7407 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2 (DEFAULT_TABLE table=test-table [id=3b094e77e91340e797c9c474a843e276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:52.436770 7407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:52.436968 7850 tablet_bootstrap.cc:654] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:52.438424 7851 tablet_bootstrap.cc:492] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98: Bootstrap starting.
I20260926 11:35:52.438683 7850 tablet_bootstrap.cc:492] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: No bootstrap required, opened a new log
I20260926 11:35:52.438783 7850 ts_tablet_manager.cc:1397] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:52.439432 7853 tablet_bootstrap.cc:492] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152: Bootstrap starting.
I20260926 11:35:52.439350 7850 raft_consensus.cc:348] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } }
I20260926 11:35:52.439512 7850 raft_consensus.cc:374] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:52.439524 7851 tablet_bootstrap.cc:654] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:52.439622 7850 raft_consensus.cc:729] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8526fc109694c32a64b01e2ce5c2331, State: Initialized, Role: FOLLOWER
I20260926 11:35:52.439827 7850 consensus_queue.cc:260] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } }
I20260926 11:35:52.440289 7850 ts_tablet_manager.cc:1428] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:52.440752 7853 tablet_bootstrap.cc:654] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:52.440949 7851 tablet_bootstrap.cc:492] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98: No bootstrap required, opened a new log
I20260926 11:35:52.441102 7851 ts_tablet_manager.cc:1397] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:52.441629 7851 raft_consensus.cc:348] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } }
I20260926 11:35:52.441788 7851 raft_consensus.cc:374] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:52.441826 7851 raft_consensus.cc:729] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4b1009d5894659b17fc8e09f3f7e98, State: Initialized, Role: FOLLOWER
I20260926 11:35:52.442052 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.440645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:35423)
0926 11:35:52.441834 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.441907 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":126,"threads_started":1}
I20260926 11:35:52.442014 7851 consensus_queue.cc:260] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } }
I20260926 11:35:52.442394 7851 ts_tablet_manager.cc:1428] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:52.442997 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.442543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:44069)
0926 11:35:52.442840 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.442893 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
I20260926 11:35:52.444253 7747 heartbeater.cc:503] Master 127.0.98.60:41515 was elected leader, sending a full tablet report...
I20260926 11:35:52.444427 7853 tablet_bootstrap.cc:492] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152: No bootstrap required, opened a new log
I20260926 11:35:52.444551 7853 ts_tablet_manager.cc:1397] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 11:35:52.445096 7853 raft_consensus.cc:348] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } }
I20260926 11:35:52.445223 7853 raft_consensus.cc:374] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:52.445259 7853 raft_consensus.cc:729] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4352e835c189494db1abdfadc82db152, State: Initialized, Role: FOLLOWER
I20260926 11:35:52.445421 7853 consensus_queue.cc:260] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } }
I20260926 11:35:52.445815 7853 ts_tablet_manager.cc:1428] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:52.446508 7611 heartbeater.cc:503] Master 127.0.98.60:41515 was elected leader, sending a full tablet report...
I20260926 11:35:52.447392 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.442720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:60113)
0926 11:35:52.447172 (+ 4452us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.447262 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4360,"thread_start_us":93,"threads_started":1}
I20260926 11:35:52.447432 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.446521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:37841)
0926 11:35:52.447230 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.447299 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":116,"threads_started":1}
I20260926 11:35:52.448424 7474 heartbeater.cc:503] Master 127.0.98.60:41515 was elected leader, sending a full tablet report...
I20260926 11:35:52.448848 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.448475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:33095)
0926 11:35:52.448680 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.448752 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 11:35:52.449376 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.448113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:47987)
0926 11:35:52.449254 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.449308 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1071}
W20260926 11:35:52.595489 7749 tablet.cc:2369] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:52.616003 7613 tablet.cc:2369] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:52.644711 7476 tablet.cc:2369] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:35:52.658098 7854 raft_consensus.cc:482] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:52.658316 7854 raft_consensus.cc:504] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } }
I20260926 11:35:52.660367 7854 leader_election.cc:288] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a4b1009d5894659b17fc8e09f3f7e98 (127.0.98.2:39887), 4352e835c189494db1abdfadc82db152 (127.0.98.1:34257)
I20260926 11:35:52.665701 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.660654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39887 (local address 127.0.98.3:38127)
0926 11:35:52.660888 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.660908 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:52.661045 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:52.661732 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:52.661735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:52.661749 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:52.661875 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.661883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.662709 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.662716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.664242 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.664257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.665292 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.665295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.665401 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.665420 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 11:35:52.665484 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 11:35:52.665542 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":35}
I20260926 11:35:52.665740 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.660987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53327 (local address 127.0.98.1:34257)
0926 11:35:52.661138 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:52.661145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:52.661168 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:35:52.661364 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:52.661367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:52.661433 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:52.661530 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:52.661933 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.662801 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.664485 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.664949 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.665090 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.665179 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:52.665456 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 11:35:52.665588 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:35:52.665707 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.660712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34257 (local address 127.0.98.3:53327)
0926 11:35:52.661047 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.661065 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:52.661214 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:52.661547 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:52.661551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:52.661568 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:52.661766 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.661774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.662939 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.662942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.664247 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:52.664257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.665100 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.665104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:52.665235 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.665265 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 11:35:52.665359 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 11:35:52.665472 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":27}
I20260926 11:35:52.666402 7427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd2b4fa1762403d9bb8c63ee8a4aee2" candidate_uuid: "f8526fc109694c32a64b01e2ce5c2331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4352e835c189494db1abdfadc82db152" is_pre_election: true
I20260926 11:35:52.666482 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.661314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38127 (local address 127.0.98.2:39887)
0926 11:35:52.661482 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:52.661487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:52.661505 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:35:52.661560 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:52.661564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:52.661615 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:52.661700 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:52.662011 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.662584 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.664432 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.665158 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.665277 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.665355 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:52.666208 (+ 853us) server_negotiation.cc:299] Negotiation successful
0926 11:35:52.666341 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 11:35:52.666579 7427 raft_consensus.cc:2454] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8526fc109694c32a64b01e2ce5c2331 in term 0.
I20260926 11:35:52.667083 7636 leader_election.cc:302] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4352e835c189494db1abdfadc82db152, f8526fc109694c32a64b01e2ce5c2331; no voters:
I20260926 11:35:52.667176 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd2b4fa1762403d9bb8c63ee8a4aee2" candidate_uuid: "f8526fc109694c32a64b01e2ce5c2331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" is_pre_election: true
I20260926 11:35:52.667446 7564 raft_consensus.cc:2454] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8526fc109694c32a64b01e2ce5c2331 in term 0.
I20260926 11:35:52.667459 7854 raft_consensus.cc:2781] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:52.667520 7854 raft_consensus.cc:482] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:52.667548 7854 raft_consensus.cc:3037] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:52.669221 7854 raft_consensus.cc:504] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } }
I20260926 11:35:52.669667 7854 leader_election.cc:288] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [CANDIDATE]: Term 1 election: Requested vote from peers 6a4b1009d5894659b17fc8e09f3f7e98 (127.0.98.2:39887), 4352e835c189494db1abdfadc82db152 (127.0.98.1:34257)
I20260926 11:35:52.670138 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd2b4fa1762403d9bb8c63ee8a4aee2" candidate_uuid: "f8526fc109694c32a64b01e2ce5c2331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4b1009d5894659b17fc8e09f3f7e98"
I20260926 11:35:52.670164 7427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd2b4fa1762403d9bb8c63ee8a4aee2" candidate_uuid: "f8526fc109694c32a64b01e2ce5c2331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4352e835c189494db1abdfadc82db152"
I20260926 11:35:52.670284 7564 raft_consensus.cc:3037] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:52.670336 7427 raft_consensus.cc:3037] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:52.671487 7427 raft_consensus.cc:2454] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8526fc109694c32a64b01e2ce5c2331 in term 1.
I20260926 11:35:52.671509 7564 raft_consensus.cc:2454] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8526fc109694c32a64b01e2ce5c2331 in term 1.
I20260926 11:35:52.671953 7634 leader_election.cc:302] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4352e835c189494db1abdfadc82db152, 6a4b1009d5894659b17fc8e09f3f7e98, f8526fc109694c32a64b01e2ce5c2331; no voters:
I20260926 11:35:52.672223 7854 raft_consensus.cc:2781] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:52.672525 7854 raft_consensus.cc:686] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 1 LEADER]: Becoming Leader. State: Replica: f8526fc109694c32a64b01e2ce5c2331, State: Running, Role: LEADER
I20260926 11:35:52.672919 7854 consensus_queue.cc:237] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } }
I20260926 11:35:52.673676 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.673277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:44097)
0926 11:35:52.673474 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.673552 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 11:35:52.676339 7296 catalog_manager.cc:5495] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8526fc109694c32a64b01e2ce5c2331 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f8526fc109694c32a64b01e2ce5c2331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8526fc109694c32a64b01e2ce5c2331" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:52.738581 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.733591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.0.98.3:45343)
0926 11:35:52.733768 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:52.733772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:52.733790 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:35:52.733896 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:52.733898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:52.733982 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:52.734132 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:52.734482 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.735187 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.736376 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:52.736777 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:52.736915 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:52.737439 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:52.738129 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:52.738287 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:52.738335 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 11:35:52.738414 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:35:52.743162 7427 raft_consensus.cc:1261] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152 [term 1 FOLLOWER]: Refusing update from remote peer f8526fc109694c32a64b01e2ce5c2331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:52.743542 7564 raft_consensus.cc:1261] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 1 FOLLOWER]: Refusing update from remote peer f8526fc109694c32a64b01e2ce5c2331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:52.744091 7863 consensus_queue.cc:1035] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4352e835c189494db1abdfadc82db152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:52.744426 7854 consensus_queue.cc:1035] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4b1009d5894659b17fc8e09f3f7e98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:52.746862 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.746433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:39219)
0926 11:35:52.746653 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.746757 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 11:35:52.747066 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.746724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:40013)
0926 11:35:52.746884 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.746955 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 11:35:52.751487 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.751077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:46533)
0926 11:35:52.751287 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.751372 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 11:35:52.752331 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.751971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:55307)
0926 11:35:52.752148 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.752228 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 11:35:52.755395 7866 mvcc.cc:205] Tried to move back new op lower bound from 7333570776027410432 to 7333570775751974912. Current Snapshot: MvccSnapshot[applied={T|T < 7333570776027410432 or (T in {7333570776027410432})}]
I20260926 11:35:52.758061 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.756097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:58431)
0926 11:35:52.757902 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.757984 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1730}
I20260926 11:35:52.758783 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.758450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:57023)
0926 11:35:52.758590 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.758676 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 11:35:52.766090 7296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:52.766462 7296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:52.772799 7296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3865d02fa61c4cfea5debb15f877ed27: Sending DeleteTablet for 3 replicas of tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2
I20260926 11:35:52.774096 7544 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:52 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51436
I20260926 11:35:52.774375 7407 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:52 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53744
I20260926 11:35:52.774735 7680 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:52 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37406
I20260926 11:35:52.774966 7877 tablet_replica.cc:330] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 4352e835c189494db1abdfadc82db152: stopping tablet replica
I20260926 11:35:52.774969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 7255
I20260926 11:35:52.775822 7876 tablet_replica.cc:330] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98: stopping tablet replica
I20260926 11:35:52.776024 7876 raft_consensus.cc:2229] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:52.776216 7876 raft_consensus.cc:2258] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:52.779953 7878 tablet_replica.cc:330] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: stopping tablet replica
I20260926 11:35:52.780160 7878 raft_consensus.cc:2229] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:52.780623 7878 raft_consensus.cc:2258] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:52.781520 7878 ts_tablet_manager.cc:1905] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:52.784657 7878 ts_tablet_manager.cc:1918] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:52.784657 7876 ts_tablet_manager.cc:1905] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P 6a4b1009d5894659b17fc8e09f3f7e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:52.784946 7878 log.cc:1196] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/5cd2b4fa1762403d9bb8c63ee8a4aee2
I20260926 11:35:52.785424 7878 ts_tablet_manager.cc:1939] T 5cd2b4fa1762403d9bb8c63ee8a4aee2 P f8526fc109694c32a64b01e2ce5c2331: Deleting consensus metadata
W20260926 11:35:52.786254 7848 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:52.784981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34257 (local address 127.0.0.1:53750)
0926 11:35:52.785182 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.785230 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:52.785330 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:52.786118 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118}
W20260926 11:35:52.786605 7281 catalog_manager.cc:4584] TS 4352e835c189494db1abdfadc82db152 (127.0.98.1:34257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 11:35:52.786830 7281 catalog_manager.cc:4841] TS f8526fc109694c32a64b01e2ce5c2331 (127.0.98.3:45343): tablet 5cd2b4fa1762403d9bb8c63ee8a4aee2 (table test-table [id=3b094e77e91340e797c9c474a843e276]) successfully deleted
I20260926 11:35:52.787505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 7481
I20260926 11:35:52.797403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 7617
I20260926 11:35:52.797726 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:52.797342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39887 (local address 127.0.0.1:51446)
0926 11:35:52.797505 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:52.797584 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 11:35:52.806047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 6973
I20260926 11:35:52.813086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 7077
2026-09-26T11:35:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7911 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:52 dist-test-slave-4pw5 krb5kdc[7881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:52 dist-test-slave-4pw5 krb5kdc[7881](info): set up 2 sockets
Sep 26 11:35:52 dist-test-slave-4pw5 krb5kdc[7881](info): commencing operation
krb5kdc: starting...
W20260926 11:35:53.953785 392 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:54 dist-test-slave-4pw5 krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 11:35:54.105014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:54.231907 7894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:54.232271 7894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:54.232348 7894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:54.237128 7894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:54.237264 7894 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 11:35:54.237298 7894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:54.237334 7894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:54.237394 7894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:54.237437 7894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:54.244871 7894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43199
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:54.246212 7894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:54.247381 7894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:54.258078 7900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:54.258231 7894 server_base.cc:1034] running on GCE node
W20260926 11:35:54.258064 7902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:54.258064 7899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:54.259474 7894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:54.261265 7894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:54.262742 7894 hybrid_clock.cc:648] HybridClock initialized: now 1790422554262703 us; error 65 us; skew 500 ppm
Sep 26 11:35:54 dist-test-slave-4pw5 krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:54.267086 7894 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 11:35:54.268573 7894 webserver.cc:460] Webserver started at http://127.0.98.62:34645/ using document root <none> and password file <none>
I20260926 11:35:54.269341 7894 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:54.269454 7894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:54.269754 7894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:54.272531 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "2eee0939d8604cffbec6c72e7145b837"
format_stamp: "Formatted at 2026-09-26 11:35:54 on dist-test-slave-4pw5"
server_key: "b6e7a83d22203abc8d51486199abe54c"
server_key_iv: "d7d0d70be9bf1ce5d199c90394921ab1"
server_key_version: "encryptionkey@0"
I20260926 11:35:54.273319 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "2eee0939d8604cffbec6c72e7145b837"
format_stamp: "Formatted at 2026-09-26 11:35:54 on dist-test-slave-4pw5"
server_key: "b6e7a83d22203abc8d51486199abe54c"
server_key_iv: "d7d0d70be9bf1ce5d199c90394921ab1"
server_key_version: "encryptionkey@0"
I20260926 11:35:54.278826 7894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:35:54.283058 7909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:54.285418 7894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:35:54.285682 7894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "2eee0939d8604cffbec6c72e7145b837"
format_stamp: "Formatted at 2026-09-26 11:35:54 on dist-test-slave-4pw5"
server_key: "b6e7a83d22203abc8d51486199abe54c"
server_key_iv: "d7d0d70be9bf1ce5d199c90394921ab1"
server_key_version: "encryptionkey@0"
I20260926 11:35:54.285861 7894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:54.315382 7894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:54.319999 7894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:54.320438 7894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:54.331700 7894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43199
I20260926 11:35:54.331799 7971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43199 every 8 connection(s)
I20260926 11:35:54.333056 7894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:54.334702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 7894
I20260926 11:35:54.334885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:54.335247 392 external_mini_cluster.cc:1428] Setting key 9ccd8217080a1096a77b624bb381cf66
I20260926 11:35:54.337029 7972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:54.344568 7972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: Bootstrap starting.
Sep 26 11:35:54 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:54.347368 7972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:54.348261 7972 log.cc:824] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:54.351166 7972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: No bootstrap required, opened a new log
I20260926 11:35:54.354912 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:54.337271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.0.98.62:43199)
0926 11:35:54.337850 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:54.337874 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:54.337929 (+ 55us) server_negotiation.cc:407] Connection header received
0926 11:35:54.338921 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:54.338945 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:54.339442 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:54.339913 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:54.340797 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.342317 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.343395 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.344147 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.344335 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:54.347900 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:54.347909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:54.347934 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:54.347984 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:54.351170 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.351823 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.351831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.351837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.351981 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.352304 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.352309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.352311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.352864 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:54.353139 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:54.353414 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 11:35:54.353834 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":165,"threads_started":1}
I20260926 11:35:54.355274 7972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee0939d8604cffbec6c72e7145b837" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43199 } }
I20260926 11:35:54.355674 7972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:54.355818 7972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eee0939d8604cffbec6c72e7145b837, State: Initialized, Role: FOLLOWER
I20260926 11:35:54.356475 7972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee0939d8604cffbec6c72e7145b837" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43199 } }
I20260926 11:35:54.356637 7972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:54.356721 7972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:54.358110 7972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:54.359282 7972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee0939d8604cffbec6c72e7145b837" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43199 } }
I20260926 11:35:54.359807 7972 leader_election.cc:302] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eee0939d8604cffbec6c72e7145b837; no voters:
I20260926 11:35:54.360204 7972 leader_election.cc:288] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:54.360519 7977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:54.360834 7977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [term 1 LEADER]: Becoming Leader. State: Replica: 2eee0939d8604cffbec6c72e7145b837, State: Running, Role: LEADER
I20260926 11:35:54.361296 7977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee0939d8604cffbec6c72e7145b837" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43199 } }
I20260926 11:35:54.361763 7972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:54.363515 7978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eee0939d8604cffbec6c72e7145b837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee0939d8604cffbec6c72e7145b837" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43199 } } }
I20260926 11:35:54.363713 7978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:54.364833 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eee0939d8604cffbec6c72e7145b837. Latest consensus state: current_term: 1 leader_uuid: "2eee0939d8604cffbec6c72e7145b837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eee0939d8604cffbec6c72e7145b837" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43199 } } }
I20260926 11:35:54.364969 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837 [sys.catalog]: This master's current role is: LEADER
W20260926 11:35:54.367259 7992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:35:54.367362 7992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:35:54.367475 7990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:54.368400 7990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:54.375479 7990 catalog_manager.cc:1347] Generated new cluster ID: a584872f93ad407f8aacd1d6017799bc
I20260926 11:35:54.375612 7990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:54.388202 7990 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:54.389405 7990 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:54.403164 7990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: Generated new TSK 0
I20260926 11:35:54.404163 7990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 11:35:54.451046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43199
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:54.577468 7998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:54.577782 7998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:54.577832 7998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:54.582229 7998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:54.582319 7998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:54.582361 7998 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:35:54.582396 7998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:54.588332 7998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:54.589622 7998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:54.590835 7998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:54.602490 8006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:54.602500 8004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:54.602490 8003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:54.603087 7998 server_base.cc:1034] running on GCE node
I20260926 11:35:54.603663 7998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:54.604394 7998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:54.605638 7998 hybrid_clock.cc:648] HybridClock initialized: now 1790422554605587 us; error 48 us; skew 500 ppm
Sep 26 11:35:54 dist-test-slave-4pw5 krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:54.608989 7998 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 11:35:54.610087 7998 webserver.cc:460] Webserver started at http://127.0.98.1:39665/ using document root <none> and password file <none>
I20260926 11:35:54.610898 7998 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:54.610982 7998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:54.611238 7998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:54.613526 7998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "fbd515b41912426eb996b9b50b03eda9"
format_stamp: "Formatted at 2026-09-26 11:35:54 on dist-test-slave-4pw5"
server_key: "214764fbee1e004d4884101d96fa9269"
server_key_iv: "c18a56d64f964c2a597a564b2e8affe8"
server_key_version: "encryptionkey@0"
I20260926 11:35:54.614122 7998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fbd515b41912426eb996b9b50b03eda9"
format_stamp: "Formatted at 2026-09-26 11:35:54 on dist-test-slave-4pw5"
server_key: "214764fbee1e004d4884101d96fa9269"
server_key_iv: "c18a56d64f964c2a597a564b2e8affe8"
server_key_version: "encryptionkey@0"
I20260926 11:35:54.618564 7998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 11:35:54.621840 8013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:54.623900 7998 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:35:54.624086 7998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "fbd515b41912426eb996b9b50b03eda9"
format_stamp: "Formatted at 2026-09-26 11:35:54 on dist-test-slave-4pw5"
server_key: "214764fbee1e004d4884101d96fa9269"
server_key_iv: "c18a56d64f964c2a597a564b2e8affe8"
server_key_version: "encryptionkey@0"
I20260926 11:35:54.624254 7998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:54.639997 7998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:54.643561 7998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:54.643898 7998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:54.645629 7998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:54.645691 7998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:54.645733 7998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:54.645749 7998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:54.657701 7998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38531
Sep 26 11:35:54 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:54.658823 7998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:54.659399 8128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38531 every 8 connection(s)
I20260926 11:35:54.660557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 7998
I20260926 11:35:54.660740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:54.661105 392 external_mini_cluster.cc:1428] Setting key 0b6d4ed1c4342a6762ae3a37bcd0b843
I20260926 11:35:54.663446 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:54.647350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39839 (local address 127.0.98.62:43199)
0926 11:35:54.647546 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:54.647551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:54.648963 (+ 1412us) server_negotiation.cc:407] Connection header received
0926 11:35:54.650595 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:54.650602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:54.650687 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:54.650823 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:54.652762 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.653486 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.655378 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.655807 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.655932 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:54.658902 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:54.658907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:54.658910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:54.658943 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:54.660992 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.661710 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.661714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.661716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.661816 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.662211 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.662215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.662216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.662467 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:54.662589 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:54.663137 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 11:35:54.663275 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 11:35:54.664486 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:54.647928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43199 (local address 127.0.98.1:39839)
0926 11:35:54.648704 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:54.648793 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:54.650241 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:54.651155 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:54.651169 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:54.651918 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:54.652567 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:54.652589 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.653627 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.653631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:54.655205 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:54.655223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.655999 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.656003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:54.656183 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:54.656893 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:54.656930 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:54.658693 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:54.661127 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:54.661136 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:54.661156 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:54.661589 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:54.661929 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:54.661933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:54.661935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:54.662094 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:54.662587 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:54.662595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:54.662885 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 11:35:54.663090 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 11:35:54.663481 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":172,"threads_started":1}
I20260926 11:35:54.666250 8129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43199
I20260926 11:35:54.666417 8129 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:54.667102 8129 heartbeater.cc:511] Master 127.0.98.62:43199 requested a full tablet report, sending...
I20260926 11:35:54.669147 7925 ts_manager.cc:194] Registered new tserver with Master: fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531)
I20260926 11:35:54.670857 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39839
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260926 11:35:54.680009 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:54.668515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36293 (local address 127.0.98.62:43199)
0926 11:35:54.671740 (+ 3225us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:54.671749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:54.671772 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:35:54.671808 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:54.671810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:54.671849 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:54.671955 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:54.672605 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.673216 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.674165 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.674552 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.674653 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:54.676215 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:54.676219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:54.676222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:54.676248 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:54.677976 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.678543 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.678547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.678548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.678640 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.679022 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.679025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.679026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.679226 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:54.679356 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:54.679753 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 11:35:54.679874 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3126}
I20260926 11:35:54.680121 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:54.668586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43199 (local address 127.0.98.1:36293)
0926 11:35:54.668876 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:54.668889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:54.668971 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:54.671979 (+ 3008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:54.671984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:54.672266 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:54.672413 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:54.672420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.673352 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.673355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:54.674037 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:54.674048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.674699 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.674701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:54.674777 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:54.675386 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:54.675415 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:54.676007 (+ 592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:54.678079 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:54.678086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:54.678091 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:54.678377 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:54.678752 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:54.678758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:54.678763 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:54.678870 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:54.679343 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:54.679347 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:54.679480 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 11:35:54.679654 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 11:35:54.679838 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":2,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 11:35:54.691922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43199
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:54.807767 8138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:54.808053 8138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:54.808102 8138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:54.812184 8138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:54.812247 8138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:54.812289 8138 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:35:54.812323 8138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:35:54.817749 8138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:54.818945 8138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:54.820114 8138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:54.830623 8144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:54.830608 8143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:54.830719 8138 server_base.cc:1034] running on GCE node
W20260926 11:35:54.830605 8146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:54.831377 8138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:54.832090 8138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:54.833269 8138 hybrid_clock.cc:648] HybridClock initialized: now 1790422554833249 us; error 40 us; skew 500 ppm
Sep 26 11:35:54 dist-test-slave-4pw5 krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:54.836397 8138 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 11:35:54.837318 8138 webserver.cc:460] Webserver started at http://127.0.98.2:36129/ using document root <none> and password file <none>
I20260926 11:35:54.837890 8138 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:54.837950 8138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:54.838171 8138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:54.840364 8138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "95364e6bf233428d9b601610ff2d4f16"
format_stamp: "Formatted at 2026-09-26 11:35:54 on dist-test-slave-4pw5"
server_key: "8047c47b7d0828940ca90a7c53d106e9"
server_key_iv: "5009aba0ed73d2f2e0e04f0a54e865ab"
server_key_version: "encryptionkey@0"
I20260926 11:35:54.840898 8138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "95364e6bf233428d9b601610ff2d4f16"
format_stamp: "Formatted at 2026-09-26 11:35:54 on dist-test-slave-4pw5"
server_key: "8047c47b7d0828940ca90a7c53d106e9"
server_key_iv: "5009aba0ed73d2f2e0e04f0a54e865ab"
server_key_version: "encryptionkey@0"
I20260926 11:35:54.844866 8138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 11:35:54.847457 8153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:54.848908 8138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:35:54.849018 8138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "95364e6bf233428d9b601610ff2d4f16"
format_stamp: "Formatted at 2026-09-26 11:35:54 on dist-test-slave-4pw5"
server_key: "8047c47b7d0828940ca90a7c53d106e9"
server_key_iv: "5009aba0ed73d2f2e0e04f0a54e865ab"
server_key_version: "encryptionkey@0"
I20260926 11:35:54.849179 8138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:54.874960 8138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:54.878264 8138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:54.878438 8138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:54.880230 8138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:54.880287 8138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:54.880330 8138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:54.880345 8138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:35:54 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:54.892207 8138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38697
I20260926 11:35:54.892278 8268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38697 every 8 connection(s)
I20260926 11:35:54.893334 8138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:35:54.895686 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:54.881301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32831 (local address 127.0.98.62:43199)
0926 11:35:54.881482 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:54.881486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:54.882551 (+ 1065us) server_negotiation.cc:407] Connection header received
0926 11:35:54.883861 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:54.883864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:54.883924 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:54.884052 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:54.885512 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.886051 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.887259 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.887690 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.887807 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:54.891276 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:54.891283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:54.891292 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:54.891320 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:54.893191 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.893836 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.893841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.893843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.893944 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.894399 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.894403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.894405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.894661 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:54.894783 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:54.895331 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 11:35:54.895472 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 11:35:54.896567 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:54.881830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43199 (local address 127.0.98.2:32831)
0926 11:35:54.882359 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:54.882402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:54.883618 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:54.884245 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:54.884252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:54.884619 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:54.885360 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:54.885376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.886181 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.886184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:54.887124 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:54.887133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.887865 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.887869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:54.888088 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:54.888767 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:54.888787 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:54.891052 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:54.893334 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:54.893341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:54.893358 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:54.893715 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:54.894073 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:54.894079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:54.894082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:54.894280 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:54.894781 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:54.894788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:54.895059 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 11:35:54.895265 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 11:35:54.895673 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":131,"threads_started":1}
I20260926 11:35:54.897938 8269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43199
I20260926 11:35:54.898152 8269 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:54.898818 8269 heartbeater.cc:511] Master 127.0.98.62:43199 requested a full tablet report, sending...
I20260926 11:35:54.899487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 8138
I20260926 11:35:54.899642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:35:54.900130 7925 ts_manager.cc:194] Registered new tserver with Master: 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697)
I20260926 11:35:54.900990 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32831
I20260926 11:35:54.901309 392 external_mini_cluster.cc:1428] Setting key aa6dee51572202be2683205679fb2cc3
I20260926 11:35:54.909525 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:54.901349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52595 (local address 127.0.98.62:43199)
0926 11:35:54.901598 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:54.901603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:54.901620 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:35:54.901657 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:54.901661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:54.901713 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:54.901800 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:54.902410 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.903113 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.904082 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.904436 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.904566 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:54.905940 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:54.905943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:54.905945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:54.905968 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:54.907650 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.908124 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.908130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.908132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.908219 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:54.908547 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:54.908550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:54.908552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:54.908795 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:54.908919 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:54.909205 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 11:35:54.909324 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:35:54.909562 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:54.900399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43199 (local address 127.0.98.2:52595)
0926 11:35:54.900844 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:54.900864 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:54.900982 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:54.901936 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:54.901940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:54.902175 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:54.902282 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:54.902289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.903255 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.903260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:54.903954 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:54.903963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:54.904591 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:54.904599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:54.904715 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:54.905404 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:54.905418 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:54.905803 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:54.907775 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:54.907781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:54.907783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:54.908016 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:54.908323 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:54.908328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:54.908330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:54.908428 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:54.908907 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:54.908910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:54.909027 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 11:35:54.909149 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 11:35:54.909325 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 11:35:54.934756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43199
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:55.063084 8278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:55.063407 8278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:55.063457 8278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:55.068696 8278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:55.068792 8278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:55.068841 8278 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:35:55.068879 8278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:35:55.075476 8278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:55.076769 8278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:55.077853 8278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:55.088701 8284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:55.088690 8286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:55.088690 8283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:55.088747 8278 server_base.cc:1034] running on GCE node
I20260926 11:35:55.089417 8278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:55.090140 8278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:55.091396 8278 hybrid_clock.cc:648] HybridClock initialized: now 1790422555091343 us; error 65 us; skew 500 ppm
Sep 26 11:35:55 dist-test-slave-4pw5 krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:55.094533 8278 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 11:35:55.095558 8278 webserver.cc:460] Webserver started at http://127.0.98.3:34481/ using document root <none> and password file <none>
I20260926 11:35:55.096189 8278 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:55.096261 8278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:55.096449 8278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:55.099231 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "e8f5db2578014a3a8637753eed3bb5dc"
format_stamp: "Formatted at 2026-09-26 11:35:55 on dist-test-slave-4pw5"
server_key: "d374a195f2b0a118d2b5b124a613fa93"
server_key_iv: "b84153e9fcf71e1bd0f335ff44f89bc0"
server_key_version: "encryptionkey@0"
I20260926 11:35:55.099932 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e8f5db2578014a3a8637753eed3bb5dc"
format_stamp: "Formatted at 2026-09-26 11:35:55 on dist-test-slave-4pw5"
server_key: "d374a195f2b0a118d2b5b124a613fa93"
server_key_iv: "b84153e9fcf71e1bd0f335ff44f89bc0"
server_key_version: "encryptionkey@0"
I20260926 11:35:55.104950 8278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 11:35:55.108938 8293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:55.110677 8278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:35:55.110813 8278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "e8f5db2578014a3a8637753eed3bb5dc"
format_stamp: "Formatted at 2026-09-26 11:35:55 on dist-test-slave-4pw5"
server_key: "d374a195f2b0a118d2b5b124a613fa93"
server_key_iv: "b84153e9fcf71e1bd0f335ff44f89bc0"
server_key_version: "encryptionkey@0"
I20260926 11:35:55.111004 8278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:55.138142 8278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:55.142102 8278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:55.142303 8278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:55.144136 8278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:55.144207 8278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:55.144254 8278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:55.144320 8278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:35:55 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:55.157565 8278 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36507
I20260926 11:35:55.157632 8408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36507 every 8 connection(s)
I20260926 11:35:55.158877 8278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:35:55.161736 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.145240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49927 (local address 127.0.98.62:43199)
0926 11:35:55.145372 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.145377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.146601 (+ 1224us) server_negotiation.cc:407] Connection header received
0926 11:35:55.148071 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.148078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.148159 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.148232 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:55.150521 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.151123 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.153408 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.153865 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.154020 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.157515 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:55.157522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:55.157525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:55.157563 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:55.159389 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:55.160129 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:55.160132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:55.160134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:55.160229 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:55.160597 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:55.160600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:55.160602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:55.160812 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:55.160922 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.161437 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.161571 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:35:55.162971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 8278
I20260926 11:35:55.163015 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.145791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43199 (local address 127.0.98.3:49927)
0926 11:35:55.146376 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.146446 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.147752 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.148756 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.148769 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.149388 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:55.150235 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.150267 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.151298 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.151305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.153226 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.153246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.154079 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.154085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.154302 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.155090 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:55.155121 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:55.157185 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:55.159666 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:55.159672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:55.159688 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:55.160005 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:55.160338 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:55.160346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:55.160348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:55.160490 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:55.160920 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:55.160927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:55.161191 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.161381 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.161732 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":113,"threads_started":1}
I20260926 11:35:55.163147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:35:55.163515 392 external_mini_cluster.cc:1428] Setting key f95e8bbfd89a8b32f89f9b0e8c39d0b9
I20260926 11:35:55.164693 8409 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43199
I20260926 11:35:55.164861 8409 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:55.165892 8409 heartbeater.cc:511] Master 127.0.98.62:43199 requested a full tablet report, sending...
I20260926 11:35:55.167557 7926 ts_manager.cc:194] Registered new tserver with Master: e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507)
I20260926 11:35:55.168455 7926 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49927
I20260926 11:35:55.176200 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.167373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40479 (local address 127.0.98.62:43199)
0926 11:35:55.167671 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.167676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.168010 (+ 334us) server_negotiation.cc:407] Connection header received
0926 11:35:55.168119 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.168121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.168244 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.168327 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:55.168902 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.169622 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.170627 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.170932 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.171023 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.172600 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:55.172603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:55.172606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:55.172632 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:55.174104 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:55.174754 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:55.174757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:55.174758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:55.174872 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:55.175197 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:55.175200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:55.175201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:55.175394 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:55.175477 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.175920 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.176041 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260926 11:35:55.176317 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.167342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43199 (local address 127.0.98.3:40479)
0926 11:35:55.167861 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.167878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.168025 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.168374 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.168376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.168624 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:55.168761 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.168769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.169769 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.169772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.170477 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.170489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.171061 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.171064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.171150 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.171941 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:55.171955 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:55.172373 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:55.174232 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:55.174235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:55.174236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:55.174472 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:55.174992 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:55.174994 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:55.174996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:55.175090 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:55.175527 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:55.175530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:55.175704 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.175813 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.176062 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":346,"threads_started":1}
I20260926 11:35:55.178087 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 11:35:55 dist-test-slave-4pw5 krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:35:55 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:35:55 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:35:55.208627 7975 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:55.201343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.0.98.62:43199)
0926 11:35:55.201507 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.201511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.201524 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:35:55.201566 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.201569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.201613 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.201716 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:55.202563 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.203160 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.204317 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.204563 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.204693 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.208501 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45814: BlockingRecv error: recv got EOF from 127.0.0.1:45814 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 11:35:55 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:55.220608 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.210301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.0.98.62:43199)
0926 11:35:55.210507 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.210514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.210977 (+ 463us) server_negotiation.cc:407] Connection header received
0926 11:35:55.211269 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.211273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.211326 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.211406 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:55.212438 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.213006 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.213958 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.214172 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.214260 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.217351 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:55.217355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:55.217357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:55.217387 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:55.218805 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:55.219290 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:55.219300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:55.219302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:55.219376 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:55.219705 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:55.219710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:55.219712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:55.219911 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:55.220067 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.220306 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.220423 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:35:55.224012 7925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:55.227916 7925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:55.245193 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.237947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36507 (local address 127.0.0.1:55730)
0926 11:35:55.238540 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.238593 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.238748 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.240027 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.240032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.240057 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:55.240250 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.240258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.241991 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.241999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.243522 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.243529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.244527 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.244531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.244650 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.244692 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.244823 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.244991 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":169,"thread_start_us":111,"threads_started":1}
I20260926 11:35:55.245311 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.238110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.0.98.3:36507)
0926 11:35:55.239065 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.239079 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.239104 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:35:55.239633 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.239639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.239855 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.240429 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:55.240578 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.241828 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.243744 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.244376 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.244525 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.244733 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.245009 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.245133 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":108,"threads_started":1}
I20260926 11:35:55.245608 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.238797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38697 (local address 127.0.0.1:33664)
0926 11:35:55.239282 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.239296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.239397 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.240164 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.240166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.240179 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:55.240317 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.240326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.242013 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.242016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.243553 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.243564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.244826 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.244831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.244969 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.244993 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.245084 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.245157 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":138,"thread_start_us":106,"threads_started":1}
I20260926 11:35:55.245850 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.238883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.0.98.1:38531)
0926 11:35:55.239364 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.239371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.239396 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:35:55.239486 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.239493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.239772 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.239973 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:55.240607 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.241429 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.243385 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.244400 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.244528 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.244715 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.245349 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.245715 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":99,"threads_started":1}
I20260926 11:35:55.246073 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.238014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38531 (local address 127.0.0.1:45776)
0926 11:35:55.238797 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.238816 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.238912 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.239959 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.239965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.239991 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:55.240469 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.240473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.241570 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.241576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.243219 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.243231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.244789 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.244794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.245064 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.245091 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.245182 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.245262 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":450,"thread_start_us":157,"threads_started":1}
I20260926 11:35:55.246533 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.239267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.0.98.2:38697)
0926 11:35:55.239698 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.239710 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.239735 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:35:55.239806 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.239810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.239989 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.240196 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:55.240464 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.241878 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.243748 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.244656 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.244824 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.245007 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.246233 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.246347 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":105,"threads_started":1}
I20260926 11:35:55.248817 8343 tablet_service.cc:1467] Processing CreateTablet for tablet 851a99c9fcd34858890d929ad9bf5a68 (DEFAULT_TABLE table=test-table [id=adf4082ba1414863b4c0b0492ba83e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:55.248806 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 851a99c9fcd34858890d929ad9bf5a68 (DEFAULT_TABLE table=test-table [id=adf4082ba1414863b4c0b0492ba83e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:55.249102 8203 tablet_service.cc:1467] Processing CreateTablet for tablet 851a99c9fcd34858890d929ad9bf5a68 (DEFAULT_TABLE table=test-table [id=adf4082ba1414863b4c0b0492ba83e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:55.250003 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851a99c9fcd34858890d929ad9bf5a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:55.249995 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851a99c9fcd34858890d929ad9bf5a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:55.250084 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851a99c9fcd34858890d929ad9bf5a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:55.258134 8438 tablet_bootstrap.cc:492] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: Bootstrap starting.
I20260926 11:35:55.258924 8437 tablet_bootstrap.cc:492] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: Bootstrap starting.
I20260926 11:35:55.259948 8439 tablet_bootstrap.cc:492] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: Bootstrap starting.
I20260926 11:35:55.261700 8437 tablet_bootstrap.cc:654] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:55.261700 8438 tablet_bootstrap.cc:654] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:55.262105 8439 tablet_bootstrap.cc:654] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:55.262732 8438 log.cc:824] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:55.263252 8439 log.cc:824] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:55.263465 8437 log.cc:824] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:55.265389 8438 tablet_bootstrap.cc:492] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: No bootstrap required, opened a new log
I20260926 11:35:55.265405 8439 tablet_bootstrap.cc:492] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: No bootstrap required, opened a new log
I20260926 11:35:55.265456 8437 tablet_bootstrap.cc:492] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: No bootstrap required, opened a new log
I20260926 11:35:55.265584 8439 ts_tablet_manager.cc:1397] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:35:55.265599 8438 ts_tablet_manager.cc:1397] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 11:35:55.265599 8437 ts_tablet_manager.cc:1397] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 11:35:55.269263 8439 raft_consensus.cc:348] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } }
I20260926 11:35:55.269584 8439 raft_consensus.cc:374] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:55.269655 8439 raft_consensus.cc:729] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95364e6bf233428d9b601610ff2d4f16, State: Initialized, Role: FOLLOWER
I20260926 11:35:55.270239 8439 consensus_queue.cc:260] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } }
I20260926 11:35:55.270395 8438 raft_consensus.cc:348] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } }
I20260926 11:35:55.270710 8438 raft_consensus.cc:374] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:55.270771 8438 raft_consensus.cc:729] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f5db2578014a3a8637753eed3bb5dc, State: Initialized, Role: FOLLOWER
I20260926 11:35:55.270684 8437 raft_consensus.cc:348] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } }
I20260926 11:35:55.270937 8437 raft_consensus.cc:374] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:55.270994 8437 raft_consensus.cc:729] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd515b41912426eb996b9b50b03eda9, State: Initialized, Role: FOLLOWER
I20260926 11:35:55.271250 8438 consensus_queue.cc:260] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } }
I20260926 11:35:55.271343 8437 consensus_queue.cc:260] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } }
I20260926 11:35:55.271836 8269 heartbeater.cc:503] Master 127.0.98.62:43199 was elected leader, sending a full tablet report...
I20260926 11:35:55.272364 8438 ts_tablet_manager.cc:1428] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260926 11:35:55.272374 8437 ts_tablet_manager.cc:1428] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260926 11:35:55.272433 8129 heartbeater.cc:503] Master 127.0.98.62:43199 was elected leader, sending a full tablet report...
I20260926 11:35:55.273048 8409 heartbeater.cc:503] Master 127.0.98.62:43199 was elected leader, sending a full tablet report...
I20260926 11:35:55.273500 8439 ts_tablet_manager.cc:1428] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260926 11:35:55.278517 8444 raft_consensus.cc:482] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:55.278788 8444 raft_consensus.cc:504] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } }
I20260926 11:35:55.280221 8444 leader_election.cc:288] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531), 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697)
I20260926 11:35:55.284246 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.280462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38697 (local address 127.0.98.3:44893)
0926 11:35:55.280734 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.280752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.280852 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.281122 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.281125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.281145 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:55.281278 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.281284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.282258 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.282263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.283069 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.283078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.283813 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.283817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.283923 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.283950 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.284023 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.284100 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260926 11:35:55.284246 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.280588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44893 (local address 127.0.98.2:38697)
0926 11:35:55.280711 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.280716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.280801 (+ 85us) server_negotiation.cc:407] Connection header received
0926 11:35:55.280963 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.280966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.281006 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.281071 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:55.281391 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.282155 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.283191 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.283680 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.283811 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.283931 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.284036 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.284100 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 11:35:55.284950 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851a99c9fcd34858890d929ad9bf5a68" candidate_uuid: "e8f5db2578014a3a8637753eed3bb5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95364e6bf233428d9b601610ff2d4f16" is_pre_election: true
I20260926 11:35:55.285262 8223 raft_consensus.cc:2454] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f5db2578014a3a8637753eed3bb5dc in term 0.
I20260926 11:35:55.285754 8295 leader_election.cc:302] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95364e6bf233428d9b601610ff2d4f16, e8f5db2578014a3a8637753eed3bb5dc; no voters:
I20260926 11:35:55.286015 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.280684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56401 (local address 127.0.98.1:38531)
0926 11:35:55.280806 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.280811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.281395 (+ 584us) server_negotiation.cc:407] Connection header received
0926 11:35:55.281559 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.281562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.281605 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.281668 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:55.281991 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.282596 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.284751 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.285313 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.285448 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.285526 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.285808 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.285888 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:35:55.286137 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.280482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38531 (local address 127.0.98.3:56401)
0926 11:35:55.281301 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.281313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.281430 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.281704 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.281707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.281724 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:55.281842 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.281848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.282778 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.282782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.284613 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.284624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.285537 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.285545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.285667 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.285687 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.285790 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.285871 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":437,"thread_start_us":163,"threads_started":1}
I20260926 11:35:55.286278 8444 raft_consensus.cc:2781] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:55.286384 8444 raft_consensus.cc:482] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:55.286456 8444 raft_consensus.cc:3037] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:55.286804 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851a99c9fcd34858890d929ad9bf5a68" candidate_uuid: "e8f5db2578014a3a8637753eed3bb5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd515b41912426eb996b9b50b03eda9" is_pre_election: true
I20260926 11:35:55.287122 8083 raft_consensus.cc:2454] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f5db2578014a3a8637753eed3bb5dc in term 0.
I20260926 11:35:55.288053 8444 raft_consensus.cc:504] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } }
I20260926 11:35:55.288498 8444 leader_election.cc:288] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [CANDIDATE]: Term 1 election: Requested vote from peers fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531), 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697)
I20260926 11:35:55.288899 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851a99c9fcd34858890d929ad9bf5a68" candidate_uuid: "e8f5db2578014a3a8637753eed3bb5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd515b41912426eb996b9b50b03eda9"
I20260926 11:35:55.289007 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851a99c9fcd34858890d929ad9bf5a68" candidate_uuid: "e8f5db2578014a3a8637753eed3bb5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95364e6bf233428d9b601610ff2d4f16"
I20260926 11:35:55.289074 8083 raft_consensus.cc:3037] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:55.289196 8223 raft_consensus.cc:3037] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:55.290462 8083 raft_consensus.cc:2454] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f5db2578014a3a8637753eed3bb5dc in term 1.
I20260926 11:35:55.290931 8296 leader_election.cc:302] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8f5db2578014a3a8637753eed3bb5dc, fbd515b41912426eb996b9b50b03eda9; no voters:
I20260926 11:35:55.291131 8444 raft_consensus.cc:2781] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:55.291450 8444 raft_consensus.cc:686] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 1 LEADER]: Becoming Leader. State: Replica: e8f5db2578014a3a8637753eed3bb5dc, State: Running, Role: LEADER
I20260926 11:35:55.291635 8223 raft_consensus.cc:2454] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f5db2578014a3a8637753eed3bb5dc in term 1.
I20260926 11:35:55.291890 8444 consensus_queue.cc:237] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } }
I20260926 11:35:55.295683 7925 catalog_manager.cc:5495] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e8f5db2578014a3a8637753eed3bb5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:55.313954 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.309614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.0.98.3:36507)
0926 11:35:55.309780 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.309784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.309798 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:35:55.310009 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.310012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.310078 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.310182 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:55.310463 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.311085 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.312268 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.312495 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.312601 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.312987 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:55.313528 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:55.313650 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.313755 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.313817 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:35:55.321493 8223 raft_consensus.cc:1261] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Refusing update from remote peer e8f5db2578014a3a8637753eed3bb5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:55.321493 8083 raft_consensus.cc:1261] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Refusing update from remote peer e8f5db2578014a3a8637753eed3bb5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:55.322335 8444 consensus_queue.cc:1035] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:55.322700 8447 consensus_queue.cc:1035] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:55.344208 7922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:55.344416 7922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:55.346917 7922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: Sending DeleteTablet for 3 replicas of tablet 851a99c9fcd34858890d929ad9bf5a68
I20260926 11:35:55.347888 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet 851a99c9fcd34858890d929ad9bf5a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55730
I20260926 11:35:55.347867 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet 851a99c9fcd34858890d929ad9bf5a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45776
I20260926 11:35:55.347862 8203 tablet_service.cc:1514] Processing DeleteTablet for tablet 851a99c9fcd34858890d929ad9bf5a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33664
I20260926 11:35:55.348429 8460 tablet_replica.cc:330] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: stopping tablet replica
I20260926 11:35:55.348837 8460 raft_consensus.cc:2229] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:55.349298 8460 raft_consensus.cc:2258] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:55.349277 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:55.349746 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:55.351822 8462 tablet_replica.cc:330] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: stopping tablet replica
I20260926 11:35:55.352039 8460 ts_tablet_manager.cc:1905] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:55.352165 8462 raft_consensus.cc:2229] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:55.352618 8462 raft_consensus.cc:2258] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:55.353040 8461 tablet_replica.cc:330] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: stopping tablet replica
I20260926 11:35:55.353355 8461 raft_consensus.cc:2229] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:55.353672 8461 raft_consensus.cc:2258] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:55.355403 8461 ts_tablet_manager.cc:1905] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:55.355547 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 264a9023e8c44f078f6d9c935d012151 (DEFAULT_TABLE table=test-table [id=2b554349e384422da9738db36156ea36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:55.355980 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264a9023e8c44f078f6d9c935d012151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:55.356194 8343 tablet_service.cc:1467] Processing CreateTablet for tablet 264a9023e8c44f078f6d9c935d012151 (DEFAULT_TABLE table=test-table [id=2b554349e384422da9738db36156ea36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:55.356220 8203 tablet_service.cc:1467] Processing CreateTablet for tablet 264a9023e8c44f078f6d9c935d012151 (DEFAULT_TABLE table=test-table [id=2b554349e384422da9738db36156ea36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:55.356474 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264a9023e8c44f078f6d9c935d012151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:55.356493 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264a9023e8c44f078f6d9c935d012151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:55.358584 8437 tablet_bootstrap.cc:492] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: Bootstrap starting.
I20260926 11:35:55.359123 8439 tablet_bootstrap.cc:492] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: Bootstrap starting.
I20260926 11:35:55.359927 8437 tablet_bootstrap.cc:654] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:55.360051 8439 tablet_bootstrap.cc:654] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:55.361253 8437 tablet_bootstrap.cc:492] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: No bootstrap required, opened a new log
I20260926 11:35:55.361358 8437 ts_tablet_manager.cc:1397] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 11:35:55.361938 8437 raft_consensus.cc:348] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.362082 8437 raft_consensus.cc:374] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:55.362126 8437 raft_consensus.cc:729] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd515b41912426eb996b9b50b03eda9, State: Initialized, Role: FOLLOWER
I20260926 11:35:55.362386 8438 tablet_bootstrap.cc:492] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: Bootstrap starting.
I20260926 11:35:55.363024 8437 consensus_queue.cc:260] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.363361 8437 ts_tablet_manager.cc:1428] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 11:35:55.363385 8438 tablet_bootstrap.cc:654] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:55.362363 8462 ts_tablet_manager.cc:1905] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:55.364527 8460 ts_tablet_manager.cc:1918] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:55.364719 8460 log.cc:1196] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/851a99c9fcd34858890d929ad9bf5a68
I20260926 11:35:55.365168 8460 ts_tablet_manager.cc:1939] T 851a99c9fcd34858890d929ad9bf5a68 P 95364e6bf233428d9b601610ff2d4f16: Deleting consensus metadata
I20260926 11:35:55.366780 7911 catalog_manager.cc:4841] TS 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697): tablet 851a99c9fcd34858890d929ad9bf5a68 (table test-table [id=adf4082ba1414863b4c0b0492ba83e8d]) successfully deleted
I20260926 11:35:55.368180 8462 ts_tablet_manager.cc:1918] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:55.368294 8462 log.cc:1196] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/851a99c9fcd34858890d929ad9bf5a68
I20260926 11:35:55.368611 8462 ts_tablet_manager.cc:1939] T 851a99c9fcd34858890d929ad9bf5a68 P e8f5db2578014a3a8637753eed3bb5dc: Deleting consensus metadata
I20260926 11:35:55.368585 8461 ts_tablet_manager.cc:1918] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:55.368709 8461 log.cc:1196] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/851a99c9fcd34858890d929ad9bf5a68
I20260926 11:35:55.369391 8461 ts_tablet_manager.cc:1939] T 851a99c9fcd34858890d929ad9bf5a68 P fbd515b41912426eb996b9b50b03eda9: Deleting consensus metadata
I20260926 11:35:55.369863 7911 catalog_manager.cc:4841] TS e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507): tablet 851a99c9fcd34858890d929ad9bf5a68 (table test-table [id=adf4082ba1414863b4c0b0492ba83e8d]) successfully deleted
I20260926 11:35:55.370637 7912 catalog_manager.cc:4841] TS fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531): tablet 851a99c9fcd34858890d929ad9bf5a68 (table test-table [id=adf4082ba1414863b4c0b0492ba83e8d]) successfully deleted
I20260926 11:35:55.371438 8439 tablet_bootstrap.cc:492] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: No bootstrap required, opened a new log
I20260926 11:35:55.371541 8439 ts_tablet_manager.cc:1397] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20260926 11:35:55.372007 8438 tablet_bootstrap.cc:492] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: No bootstrap required, opened a new log
I20260926 11:35:55.372081 8438 ts_tablet_manager.cc:1397] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260926 11:35:55.372063 8439 raft_consensus.cc:348] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.372184 8439 raft_consensus.cc:374] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:55.372216 8439 raft_consensus.cc:729] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95364e6bf233428d9b601610ff2d4f16, State: Initialized, Role: FOLLOWER
I20260926 11:35:55.372340 8439 consensus_queue.cc:260] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.372586 8439 ts_tablet_manager.cc:1428] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:35:55.372555 8438 raft_consensus.cc:348] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.372673 8438 raft_consensus.cc:374] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:55.372707 8438 raft_consensus.cc:729] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f5db2578014a3a8637753eed3bb5dc, State: Initialized, Role: FOLLOWER
I20260926 11:35:55.372845 8438 consensus_queue.cc:260] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.373152 8438 ts_tablet_manager.cc:1428] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260926 11:35:55.394827 8270 tablet.cc:2369] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:55.410142 8410 tablet.cc:2369] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:35:55.410822 8130 tablet.cc:2369] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:35:55.588800 8443 raft_consensus.cc:482] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:55.588990 8443 raft_consensus.cc:504] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.590459 8443 leader_election.cc:288] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531), e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507)
I20260926 11:35:55.594931 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.590876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52825 (local address 127.0.98.1:38531)
0926 11:35:55.591040 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.591047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.591636 (+ 589us) server_negotiation.cc:407] Connection header received
0926 11:35:55.591781 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.591784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.591830 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.591912 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:55.592238 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.592818 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.593857 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.594241 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.594355 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.594421 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.594688 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.594772 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:35:55.594960 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.590868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59505 (local address 127.0.98.3:36507)
0926 11:35:55.591020 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.591025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.591344 (+ 319us) server_negotiation.cc:407] Connection header received
0926 11:35:55.591502 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.591506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.591606 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.591697 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:55.592023 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.592651 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.593822 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.594243 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.594376 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.594447 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.594697 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.594779 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:35:55.594980 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.590779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36507 (local address 127.0.98.2:59505)
0926 11:35:55.591189 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.591202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.591362 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.591712 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.591733 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.591757 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:55.591901 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.591911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.592776 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.592779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.593681 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.593690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.594398 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.594402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.594528 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.594572 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.594675 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.594795 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":63,"thread_start_us":96,"threads_started":1}
I20260926 11:35:55.594978 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.590811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38531 (local address 127.0.98.2:52825)
0926 11:35:55.591503 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.591518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.591669 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.591950 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.591952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.591992 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:55.592116 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.592124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.592938 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.592941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.593700 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.593708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.594383 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.594388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.594537 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.594574 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.594675 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.594800 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":32,"thread_start_us":75,"threads_started":1}
I20260926 11:35:55.595548 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264a9023e8c44f078f6d9c935d012151" candidate_uuid: "95364e6bf233428d9b601610ff2d4f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f5db2578014a3a8637753eed3bb5dc" is_pre_election: true
I20260926 11:35:55.595901 8363 raft_consensus.cc:2454] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95364e6bf233428d9b601610ff2d4f16 in term 0.
I20260926 11:35:55.595907 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264a9023e8c44f078f6d9c935d012151" candidate_uuid: "95364e6bf233428d9b601610ff2d4f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd515b41912426eb996b9b50b03eda9" is_pre_election: true
I20260926 11:35:55.596047 8083 raft_consensus.cc:2454] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95364e6bf233428d9b601610ff2d4f16 in term 0.
I20260926 11:35:55.596412 8155 leader_election.cc:302] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 95364e6bf233428d9b601610ff2d4f16, e8f5db2578014a3a8637753eed3bb5dc, fbd515b41912426eb996b9b50b03eda9; no voters:
I20260926 11:35:55.596732 8443 raft_consensus.cc:2781] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:55.596793 8443 raft_consensus.cc:482] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:55.596820 8443 raft_consensus.cc:3037] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:55.598014 8443 raft_consensus.cc:504] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.598462 8443 leader_election.cc:288] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [CANDIDATE]: Term 1 election: Requested vote from peers fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531), e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507)
I20260926 11:35:55.598922 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264a9023e8c44f078f6d9c935d012151" candidate_uuid: "95364e6bf233428d9b601610ff2d4f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd515b41912426eb996b9b50b03eda9"
I20260926 11:35:55.599066 8083 raft_consensus.cc:3037] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:55.599030 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264a9023e8c44f078f6d9c935d012151" candidate_uuid: "95364e6bf233428d9b601610ff2d4f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f5db2578014a3a8637753eed3bb5dc"
I20260926 11:35:55.599155 8363 raft_consensus.cc:3037] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:55.600389 8363 raft_consensus.cc:2454] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95364e6bf233428d9b601610ff2d4f16 in term 1.
I20260926 11:35:55.600394 8083 raft_consensus.cc:2454] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95364e6bf233428d9b601610ff2d4f16 in term 1.
I20260926 11:35:55.600735 8156 leader_election.cc:302] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95364e6bf233428d9b601610ff2d4f16, fbd515b41912426eb996b9b50b03eda9; no voters:
I20260926 11:35:55.600915 8443 raft_consensus.cc:2781] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:55.601122 8443 raft_consensus.cc:686] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 1 LEADER]: Becoming Leader. State: Replica: 95364e6bf233428d9b601610ff2d4f16, State: Running, Role: LEADER
I20260926 11:35:55.601392 8443 consensus_queue.cc:237] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.603825 7922 catalog_manager.cc:5495] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "95364e6bf233428d9b601610ff2d4f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } health_report { overall_health: UNKNOWN } } }
Sep 26 11:35:55 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422554, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:55.701745 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.690121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43199 (local address 127.0.0.1:45822)
0926 11:35:55.690570 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.690587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.690750 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.691108 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.691110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.691904 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:55.692037 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.692044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.693100 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.693103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.693775 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.693784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.694345 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.694348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.694425 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.695269 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:55.695283 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:55.697278 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:55.699674 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:55.699687 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:55.699695 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:55.700051 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:55.700419 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:55.700422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:55.700423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:55.700514 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:55.701054 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:55.701061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:55.701242 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.701358 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.701524 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":192,"threads_started":1}
I20260926 11:35:55.702041 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.690225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45822 (local address 127.0.98.62:43199)
0926 11:35:55.690400 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.690406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.690705 (+ 299us) server_negotiation.cc:407] Connection header received
0926 11:35:55.690854 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.690857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.690913 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.691011 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:55.692318 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.692959 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.693909 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.694167 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.694289 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.697583 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:55.697588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:55.697590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:55.697630 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:55.699395 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:55.700187 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:55.700191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:55.700192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:55.700305 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:55.700678 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:55.700681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:55.700682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:55.700911 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:55.701014 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.701570 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.701722 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 11:35:55.707561 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45822:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 11:35:55.708597 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:55.715180 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 5a94d47b959c494481df845746be7a70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b92e129ebedf49e4b9ac27d4a777faad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:35:55.715392 8203 tablet_service.cc:1467] Processing CreateTablet for tablet 5a94d47b959c494481df845746be7a70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b92e129ebedf49e4b9ac27d4a777faad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:35:55.715711 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a94d47b959c494481df845746be7a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:55.716035 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a94d47b959c494481df845746be7a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:55.719137 8437 tablet_bootstrap.cc:492] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: Bootstrap starting.
I20260926 11:35:55.719353 8439 tablet_bootstrap.cc:492] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: Bootstrap starting.
I20260926 11:35:55.719726 8343 tablet_service.cc:1467] Processing CreateTablet for tablet 5a94d47b959c494481df845746be7a70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b92e129ebedf49e4b9ac27d4a777faad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:35:55.720084 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a94d47b959c494481df845746be7a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:55.720413 8439 tablet_bootstrap.cc:654] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:55.720417 8437 tablet_bootstrap.cc:654] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:55.722529 8437 tablet_bootstrap.cc:492] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: No bootstrap required, opened a new log
I20260926 11:35:55.722640 8437 ts_tablet_manager.cc:1397] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 11:35:55.722962 8438 tablet_bootstrap.cc:492] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: Bootstrap starting.
I20260926 11:35:55.723268 8437 raft_consensus.cc:348] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.723426 8437 raft_consensus.cc:374] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:55.723457 8437 raft_consensus.cc:729] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd515b41912426eb996b9b50b03eda9, State: Initialized, Role: FOLLOWER
I20260926 11:35:55.723711 8437 consensus_queue.cc:260] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.724153 8438 tablet_bootstrap.cc:654] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:55.724174 8439 tablet_bootstrap.cc:492] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: No bootstrap required, opened a new log
I20260926 11:35:55.724092 8445 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.724280 8439 ts_tablet_manager.cc:1397] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 11:35:55.724298 8445 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:55.724858 8439 raft_consensus.cc:348] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.724998 8439 raft_consensus.cc:374] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:55.725041 8439 raft_consensus.cc:729] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95364e6bf233428d9b601610ff2d4f16, State: Initialized, Role: FOLLOWER
I20260926 11:35:55.725175 8439 consensus_queue.cc:260] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.725479 8439 ts_tablet_manager.cc:1428] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 11:35:55.725783 8443 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.725965 8443 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:55.726186 8437 ts_tablet_manager.cc:1428] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:55.727368 8438 tablet_bootstrap.cc:492] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: No bootstrap required, opened a new log
I20260926 11:35:55.727490 8438 ts_tablet_manager.cc:1397] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260926 11:35:55.728118 8438 raft_consensus.cc:348] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.728271 8438 raft_consensus.cc:374] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:55.728313 8438 raft_consensus.cc:729] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f5db2578014a3a8637753eed3bb5dc, State: Initialized, Role: FOLLOWER
I20260926 11:35:55.728466 8438 consensus_queue.cc:260] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.728809 8438 ts_tablet_manager.cc:1428] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:35:55.728829 8455 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.729091 8455 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:55.860271 8443 raft_consensus.cc:482] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:55.860533 8443 raft_consensus.cc:504] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.861279 8443 leader_election.cc:288] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531), e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507)
I20260926 11:35:55.861953 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a94d47b959c494481df845746be7a70" candidate_uuid: "95364e6bf233428d9b601610ff2d4f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd515b41912426eb996b9b50b03eda9" is_pre_election: true
I20260926 11:35:55.861953 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a94d47b959c494481df845746be7a70" candidate_uuid: "95364e6bf233428d9b601610ff2d4f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f5db2578014a3a8637753eed3bb5dc" is_pre_election: true
I20260926 11:35:55.862174 8083 raft_consensus.cc:2454] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95364e6bf233428d9b601610ff2d4f16 in term 0.
I20260926 11:35:55.862174 8363 raft_consensus.cc:2454] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95364e6bf233428d9b601610ff2d4f16 in term 0.
I20260926 11:35:55.862556 8156 leader_election.cc:302] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95364e6bf233428d9b601610ff2d4f16, fbd515b41912426eb996b9b50b03eda9; no voters:
I20260926 11:35:55.862881 8443 raft_consensus.cc:2781] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:55.862994 8443 raft_consensus.cc:482] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:55.863044 8443 raft_consensus.cc:3037] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:55.864593 8443 raft_consensus.cc:504] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.865070 8443 leader_election.cc:288] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [CANDIDATE]: Term 1 election: Requested vote from peers fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531), e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507)
I20260926 11:35:55.865705 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a94d47b959c494481df845746be7a70" candidate_uuid: "95364e6bf233428d9b601610ff2d4f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd515b41912426eb996b9b50b03eda9"
I20260926 11:35:55.865852 8083 raft_consensus.cc:3037] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:55.866918 8083 raft_consensus.cc:2454] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95364e6bf233428d9b601610ff2d4f16 in term 1.
I20260926 11:35:55.867261 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a94d47b959c494481df845746be7a70" candidate_uuid: "95364e6bf233428d9b601610ff2d4f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f5db2578014a3a8637753eed3bb5dc"
I20260926 11:35:55.867422 8363 raft_consensus.cc:3037] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:55.867451 8156 leader_election.cc:302] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95364e6bf233428d9b601610ff2d4f16, fbd515b41912426eb996b9b50b03eda9; no voters:
I20260926 11:35:55.867816 8443 raft_consensus.cc:2781] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:55.867959 8443 raft_consensus.cc:686] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [term 1 LEADER]: Becoming Leader. State: Replica: 95364e6bf233428d9b601610ff2d4f16, State: Running, Role: LEADER
I20260926 11:35:55.868139 8443 consensus_queue.cc:237] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:55.868695 8363 raft_consensus.cc:2454] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95364e6bf233428d9b601610ff2d4f16 in term 1.
I20260926 11:35:55.869603 8468 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: TxnStatusTablet state changed. Reason: New leader 95364e6bf233428d9b601610ff2d4f16. Latest consensus state: current_term: 1 leader_uuid: "95364e6bf233428d9b601610ff2d4f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.869791 8468 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:35:55.870090 8480 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 11:35:55.870286 8480 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 11:35:55.871359 7926 catalog_manager.cc:5495] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "95364e6bf233428d9b601610ff2d4f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:55.923228 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.918144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.0.98.2:38697)
0926 11:35:55.918466 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.918471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.918489 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:35:55.918732 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.918734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.918800 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.918935 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:55.919232 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.919914 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.920837 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.921096 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.921189 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.921622 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:55.922713 (+ 1091us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:55.922853 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.922921 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.922992 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":132,"threads_started":1}
I20260926 11:35:55.923187 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.917990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38697 (local address 127.0.0.1:33676)
0926 11:35:55.918312 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.918334 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.918493 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.918959 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.918962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.918969 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:55.919105 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.919111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.920029 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.920035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.920711 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.920718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.921320 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.921326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.921392 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.921446 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:55.922819 (+ 1373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:55.922837 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.922901 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.923015 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 11:35:55.927518 8083 raft_consensus.cc:1261] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Refusing update from remote peer 95364e6bf233428d9b601610ff2d4f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:55.927929 8363 raft_consensus.cc:1261] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Refusing update from remote peer 95364e6bf233428d9b601610ff2d4f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:55.928367 8468 consensus_queue.cc:1035] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:55.928663 8443 consensus_queue.cc:1035] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:55.930371 8445 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: TxnStatusTablet state changed. Reason: New leader 95364e6bf233428d9b601610ff2d4f16. Latest consensus state: current_term: 1 leader_uuid: "95364e6bf233428d9b601610ff2d4f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.930524 8445 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:55.933697 8455 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: TxnStatusTablet state changed. Reason: New leader 95364e6bf233428d9b601610ff2d4f16. Latest consensus state: current_term: 1 leader_uuid: "95364e6bf233428d9b601610ff2d4f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.933807 8468 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95364e6bf233428d9b601610ff2d4f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.933897 8455 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:55.933940 8468 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:35:55.935436 8468 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95364e6bf233428d9b601610ff2d4f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.935635 8468 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P 95364e6bf233428d9b601610ff2d4f16: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:35:55.935631 8487 mvcc.cc:205] Tried to move back new op lower bound from 7333570789071691776 to 7333570788837236736. Current Snapshot: MvccSnapshot[applied={T|T < 7333570789071691776}]
I20260926 11:35:55.937253 8445 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95364e6bf233428d9b601610ff2d4f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.937422 8445 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P fbd515b41912426eb996b9b50b03eda9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:55.938876 8455 tablet_replica.cc:439] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95364e6bf233428d9b601610ff2d4f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } } }
I20260926 11:35:55.939023 8455 tablet_replica.cc:442] T 5a94d47b959c494481df845746be7a70 P e8f5db2578014a3a8637753eed3bb5dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:35:55.939275 8485 mvcc.cc:205] Tried to move back new op lower bound from 7333570789071691776 to 7333570788837236736. Current Snapshot: MvccSnapshot[applied={T|T < 7333570789071691776 or (T in {7333570789071691776})}]
I20260926 11:35:55.946102 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.942358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.0.98.2:38697)
0926 11:35:55.942512 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.942517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.942745 (+ 228us) server_negotiation.cc:407] Connection header received
0926 11:35:55.942924 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.942928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.942989 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.943072 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:55.943353 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.944011 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.944952 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.945201 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.945330 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.945588 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:55.945682 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:55.945776 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.945865 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.945930 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:35:55.962217 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.957303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34143 (local address 127.0.98.2:38697)
0926 11:35:55.957438 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:55.957443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:55.957614 (+ 171us) server_negotiation.cc:407] Connection header received
0926 11:35:55.957836 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:55.957840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:55.957898 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:55.957991 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:55.958347 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.958946 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.960813 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.961078 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.961216 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.961605 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:55.961705 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:55.961805 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:55.961918 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 11:35:55.961974 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":65}
I20260926 11:35:55.962217 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:55.956993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38697 (local address 127.0.98.2:34143)
0926 11:35:55.957485 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:55.957509 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:55.957712 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:55.957996 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:55.958000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:55.958007 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:55.958227 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.958234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.959064 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.959071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.960660 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:55.960681 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:55.961234 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:55.961237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:55.961357 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:55.961431 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:55.961805 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:55.961813 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 11:35:55.961889 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 11:35:55.961999 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":40,"thread_start_us":174,"threads_started":1}
I20260926 11:35:55.972697 8363 raft_consensus.cc:1261] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Refusing update from remote peer 95364e6bf233428d9b601610ff2d4f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:55.972877 8083 raft_consensus.cc:1261] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Refusing update from remote peer 95364e6bf233428d9b601610ff2d4f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:55.973223 8497 consensus_queue.cc:1035] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:55.973505 8443 consensus_queue.cc:1035] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:55.979079 8484 mvcc.cc:205] Tried to move back new op lower bound from 7333570789258895360 to 7333570787745206272. Current Snapshot: MvccSnapshot[applied={T|T < 7333570789258895360}]
I20260926 11:35:56.029959 7925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:56.030167 7925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:56.033767 7925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: Sending DeleteTablet for 3 replicas of tablet 264a9023e8c44f078f6d9c935d012151
I20260926 11:35:56.034799 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet 264a9023e8c44f078f6d9c935d012151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55730
I20260926 11:35:56.034799 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet 264a9023e8c44f078f6d9c935d012151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45776
I20260926 11:35:56.035249 8202 tablet_service.cc:1514] Processing DeleteTablet for tablet 264a9023e8c44f078f6d9c935d012151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33664
I20260926 11:35:56.035763 8504 tablet_replica.cc:330] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: stopping tablet replica
I20260926 11:35:56.035806 8505 tablet_replica.cc:330] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: stopping tablet replica
I20260926 11:35:56.035960 8504 raft_consensus.cc:2229] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:56.036120 8505 raft_consensus.cc:2229] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:56.036141 8504 raft_consensus.cc:2258] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:56.035537 8503 tablet_replica.cc:330] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: stopping tablet replica
I20260926 11:35:56.036401 8503 raft_consensus.cc:2229] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:56.036556 8503 raft_consensus.cc:2258] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:56.036811 8505 raft_consensus.cc:2258] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:56.037518 8503 ts_tablet_manager.cc:1905] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:56.037531 8505 ts_tablet_manager.cc:1905] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:56.041193 8505 ts_tablet_manager.cc:1918] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:56.041193 8503 ts_tablet_manager.cc:1918] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:56.041291 8503 log.cc:1196] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/264a9023e8c44f078f6d9c935d012151
I20260926 11:35:56.041483 8505 log.cc:1196] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/264a9023e8c44f078f6d9c935d012151
I20260926 11:35:56.041589 8503 ts_tablet_manager.cc:1939] T 264a9023e8c44f078f6d9c935d012151 P fbd515b41912426eb996b9b50b03eda9: Deleting consensus metadata
I20260926 11:35:56.041899 8505 ts_tablet_manager.cc:1939] T 264a9023e8c44f078f6d9c935d012151 P 95364e6bf233428d9b601610ff2d4f16: Deleting consensus metadata
I20260926 11:35:56.042398 8504 ts_tablet_manager.cc:1905] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:56.043672 7911 catalog_manager.cc:4841] TS 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697): tablet 264a9023e8c44f078f6d9c935d012151 (table test-table [id=2b554349e384422da9738db36156ea36]) successfully deleted
I20260926 11:35:56.044587 7912 catalog_manager.cc:4841] TS fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531): tablet 264a9023e8c44f078f6d9c935d012151 (table test-table [id=2b554349e384422da9738db36156ea36]) successfully deleted
I20260926 11:35:56.045708 8504 ts_tablet_manager.cc:1918] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:56.045838 8504 log.cc:1196] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/264a9023e8c44f078f6d9c935d012151
I20260926 11:35:56.046093 8504 ts_tablet_manager.cc:1939] T 264a9023e8c44f078f6d9c935d012151 P e8f5db2578014a3a8637753eed3bb5dc: Deleting consensus metadata
I20260926 11:35:56.046926 7911 catalog_manager.cc:4841] TS e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507): tablet 264a9023e8c44f078f6d9c935d012151 (table test-table [id=2b554349e384422da9738db36156ea36]) successfully deleted
Sep 26 11:35:56 dist-test-slave-4pw5 krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 11:35:56 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:35:56 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:35:56.072476 7975 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:35:56.063332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45834 (local address 127.0.98.62:43199)
0926 11:35:56.063534 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:56.063540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:56.064166 (+ 626us) server_negotiation.cc:407] Connection header received
0926 11:35:56.064406 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:56.064410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:56.064477 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:56.064594 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:56.065908 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.066726 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.068003 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.068369 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.068506 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.072315 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45834: BlockingRecv error: recv got EOF from 127.0.0.1:45834 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 11:35:56 dist-test-slave-4pw5 krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:35:56.086262 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.074654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.0.98.62:43199)
0926 11:35:56.074801 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:56.074806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:56.074895 (+ 89us) server_negotiation.cc:407] Connection header received
0926 11:35:56.075031 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:56.075035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:56.075091 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:56.075409 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:56.076208 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.077040 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.077978 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.078341 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.078457 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.081724 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:56.081729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:56.081733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:56.081776 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:56.083811 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:56.084381 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:56.084386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:56.084388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:56.084510 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:56.085052 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:56.085056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:56.085058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:56.085336 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:56.085502 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:56.085902 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 11:35:56.086069 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:35:56.090785 7925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:56.091456 7925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:56.098485 8343 tablet_service.cc:1467] Processing CreateTablet for tablet 0d43c263b38b428a8798bb4317de798d (DEFAULT_TABLE table=test-table [id=dc5c34a782eb43d2b42222b1e55bd911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:56.098721 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 0d43c263b38b428a8798bb4317de798d (DEFAULT_TABLE table=test-table [id=dc5c34a782eb43d2b42222b1e55bd911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:56.098892 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d43c263b38b428a8798bb4317de798d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:56.099102 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d43c263b38b428a8798bb4317de798d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:56.101699 8437 tablet_bootstrap.cc:492] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: Bootstrap starting.
I20260926 11:35:56.101960 8202 tablet_service.cc:1467] Processing CreateTablet for tablet 0d43c263b38b428a8798bb4317de798d (DEFAULT_TABLE table=test-table [id=dc5c34a782eb43d2b42222b1e55bd911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:56.102298 8202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d43c263b38b428a8798bb4317de798d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:56.102897 8437 tablet_bootstrap.cc:654] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:56.104877 8437 tablet_bootstrap.cc:492] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: No bootstrap required, opened a new log
I20260926 11:35:56.105005 8437 ts_tablet_manager.cc:1397] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 11:35:56.105559 8439 tablet_bootstrap.cc:492] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: Bootstrap starting.
I20260926 11:35:56.105546 8437 raft_consensus.cc:348] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.105726 8437 raft_consensus.cc:374] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:56.105764 8437 raft_consensus.cc:729] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd515b41912426eb996b9b50b03eda9, State: Initialized, Role: FOLLOWER
I20260926 11:35:56.105954 8437 consensus_queue.cc:260] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.106290 8437 ts_tablet_manager.cc:1428] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 11:35:56.106583 8439 tablet_bootstrap.cc:654] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:56.107259 8438 tablet_bootstrap.cc:492] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: Bootstrap starting.
I20260926 11:35:56.108192 8438 tablet_bootstrap.cc:654] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:56.109359 8439 tablet_bootstrap.cc:492] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: No bootstrap required, opened a new log
I20260926 11:35:56.109467 8438 tablet_bootstrap.cc:492] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: No bootstrap required, opened a new log
I20260926 11:35:56.109484 8439 ts_tablet_manager.cc:1397] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260926 11:35:56.109553 8438 ts_tablet_manager.cc:1397] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:35:56.110018 8438 raft_consensus.cc:348] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.110136 8438 raft_consensus.cc:374] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:56.110172 8438 raft_consensus.cc:729] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f5db2578014a3a8637753eed3bb5dc, State: Initialized, Role: FOLLOWER
I20260926 11:35:56.110100 8439 raft_consensus.cc:348] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.110247 8439 raft_consensus.cc:374] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:56.110291 8439 raft_consensus.cc:729] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95364e6bf233428d9b601610ff2d4f16, State: Initialized, Role: FOLLOWER
I20260926 11:35:56.110306 8438 consensus_queue.cc:260] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.110431 8439 consensus_queue.cc:260] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.110584 8438 ts_tablet_manager.cc:1428] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 11:35:56.110771 8439 ts_tablet_manager.cc:1428] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:35:56.251999 8445 raft_consensus.cc:482] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:56.252420 8445 raft_consensus.cc:504] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.253850 8445 leader_election.cc:288] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697), e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507)
I20260926 11:35:56.260046 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.254586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57589 (local address 127.0.98.2:38697)
0926 11:35:56.254806 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:56.254812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:56.255062 (+ 250us) server_negotiation.cc:407] Connection header received
0926 11:35:56.255352 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:56.255359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:56.255432 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:56.255521 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:56.255957 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.256923 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.258441 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.259253 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.259420 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.259519 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:56.259798 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 11:35:56.259895 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 11:35:56.260136 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.254481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38697 (local address 127.0.98.1:57589)
0926 11:35:56.254876 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:56.254894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:56.255122 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:56.255610 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:56.255615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:56.255641 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:56.255804 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:56.255815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.257044 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.257047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:56.258284 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:56.258294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.259457 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.259461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:56.259627 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.259661 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 11:35:56.259768 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 11:35:56.259865 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":2,"thread_start_us":149,"threads_started":1}
I20260926 11:35:56.260892 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.255051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36507 (local address 127.0.98.1:51049)
0926 11:35:56.255297 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:56.255312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:56.255421 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:56.255935 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:56.255939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:56.255962 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:35:56.256628 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:56.256640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.257884 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.257887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:56.259265 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:56.259276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.260399 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.260403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:56.260532 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.260553 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:35:56.260654 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 11:35:56.260743 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":5,"thread_start_us":73,"threads_started":1}
I20260926 11:35:56.260900 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.255253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51049 (local address 127.0.98.3:36507)
0926 11:35:56.255615 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:56.255620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:56.255636 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:35:56.255690 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:56.255693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:56.255744 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:56.255863 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:35:56.256813 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.257693 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.259435 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.260218 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.260348 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.260424 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:56.260687 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 11:35:56.260769 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":117,"threads_started":1}
I20260926 11:35:56.260825 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d43c263b38b428a8798bb4317de798d" candidate_uuid: "fbd515b41912426eb996b9b50b03eda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95364e6bf233428d9b601610ff2d4f16" is_pre_election: true
I20260926 11:35:56.261080 8223 raft_consensus.cc:2454] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd515b41912426eb996b9b50b03eda9 in term 0.
I20260926 11:35:56.261544 8015 leader_election.cc:302] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95364e6bf233428d9b601610ff2d4f16, fbd515b41912426eb996b9b50b03eda9; no voters:
I20260926 11:35:56.261890 8445 raft_consensus.cc:2781] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:56.261965 8445 raft_consensus.cc:482] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:56.262019 8445 raft_consensus.cc:3037] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:56.262110 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d43c263b38b428a8798bb4317de798d" candidate_uuid: "fbd515b41912426eb996b9b50b03eda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f5db2578014a3a8637753eed3bb5dc" is_pre_election: true
I20260926 11:35:56.262454 8363 raft_consensus.cc:2454] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd515b41912426eb996b9b50b03eda9 in term 0.
I20260926 11:35:56.263731 8445 raft_consensus.cc:504] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.264169 8445 leader_election.cc:288] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [CANDIDATE]: Term 1 election: Requested vote from peers 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697), e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507)
I20260926 11:35:56.264806 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d43c263b38b428a8798bb4317de798d" candidate_uuid: "fbd515b41912426eb996b9b50b03eda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95364e6bf233428d9b601610ff2d4f16"
I20260926 11:35:56.264971 8223 raft_consensus.cc:3037] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:56.264909 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d43c263b38b428a8798bb4317de798d" candidate_uuid: "fbd515b41912426eb996b9b50b03eda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f5db2578014a3a8637753eed3bb5dc"
I20260926 11:35:56.265098 8363 raft_consensus.cc:3037] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:56.266292 8363 raft_consensus.cc:2454] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd515b41912426eb996b9b50b03eda9 in term 1.
I20260926 11:35:56.266848 8223 raft_consensus.cc:2454] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd515b41912426eb996b9b50b03eda9 in term 1.
I20260926 11:35:56.267161 8015 leader_election.cc:302] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8f5db2578014a3a8637753eed3bb5dc, fbd515b41912426eb996b9b50b03eda9; no voters:
I20260926 11:35:56.267545 8445 raft_consensus.cc:2781] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:56.267939 8445 raft_consensus.cc:686] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 1 LEADER]: Becoming Leader. State: Replica: fbd515b41912426eb996b9b50b03eda9, State: Running, Role: LEADER
I20260926 11:35:56.268332 8445 consensus_queue.cc:237] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.271773 7922 catalog_manager.cc:5495] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbd515b41912426eb996b9b50b03eda9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fbd515b41912426eb996b9b50b03eda9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:56.311081 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.306002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.0.98.1:38531)
0926 11:35:56.306370 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:56.306379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:56.306410 (+ 31us) server_negotiation.cc:407] Connection header received
0926 11:35:56.306521 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:56.306523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:56.306593 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:56.306765 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:56.307152 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.307891 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.309009 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.309374 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.309478 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.309862 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:56.310569 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:56.310691 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:56.310817 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 11:35:56.310897 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":176,"threads_started":1}
I20260926 11:35:56.315400 8223 raft_consensus.cc:1261] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Refusing update from remote peer fbd515b41912426eb996b9b50b03eda9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:56.315523 8363 raft_consensus.cc:1261] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Refusing update from remote peer fbd515b41912426eb996b9b50b03eda9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:56.316087 8445 consensus_queue.cc:1035] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:56.316458 8524 consensus_queue.cc:1035] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:56.333787 7922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:56.333993 7922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:56.336738 7922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: Sending DeleteTablet for 3 replicas of tablet 0d43c263b38b428a8798bb4317de798d
I20260926 11:35:56.337631 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d43c263b38b428a8798bb4317de798d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45776
I20260926 11:35:56.337898 8503 tablet_replica.cc:330] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: stopping tablet replica
I20260926 11:35:56.337949 8202 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d43c263b38b428a8798bb4317de798d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33664
I20260926 11:35:56.338047 8503 raft_consensus.cc:2229] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:56.338188 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d43c263b38b428a8798bb4317de798d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55730
I20260926 11:35:56.338414 8505 tablet_replica.cc:330] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: stopping tablet replica
I20260926 11:35:56.338553 8503 raft_consensus.cc:2258] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:56.338562 8505 raft_consensus.cc:2229] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:56.338725 8505 raft_consensus.cc:2258] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:56.338943 8504 tablet_replica.cc:330] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: stopping tablet replica
I20260926 11:35:56.339089 8504 raft_consensus.cc:2229] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:56.339222 8504 raft_consensus.cc:2258] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:56.340585 8505 ts_tablet_manager.cc:1905] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:56.340636 8504 ts_tablet_manager.cc:1905] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:56.344107 8505 ts_tablet_manager.cc:1918] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:56.344262 8505 log.cc:1196] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/0d43c263b38b428a8798bb4317de798d
I20260926 11:35:56.344177 8504 ts_tablet_manager.cc:1918] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:56.344527 8504 log.cc:1196] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/0d43c263b38b428a8798bb4317de798d
I20260926 11:35:56.344581 8505 ts_tablet_manager.cc:1939] T 0d43c263b38b428a8798bb4317de798d P 95364e6bf233428d9b601610ff2d4f16: Deleting consensus metadata
I20260926 11:35:56.344901 8504 ts_tablet_manager.cc:1939] T 0d43c263b38b428a8798bb4317de798d P e8f5db2578014a3a8637753eed3bb5dc: Deleting consensus metadata
I20260926 11:35:56.345815 7911 catalog_manager.cc:4841] TS 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697): tablet 0d43c263b38b428a8798bb4317de798d (table test-table [id=dc5c34a782eb43d2b42222b1e55bd911]) successfully deleted
I20260926 11:35:56.346081 7911 catalog_manager.cc:4841] TS e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507): tablet 0d43c263b38b428a8798bb4317de798d (table test-table [id=dc5c34a782eb43d2b42222b1e55bd911]) successfully deleted
I20260926 11:35:56.347260 8503 ts_tablet_manager.cc:1905] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:56.347169 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:35:56.348129 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:35:56.350486 8503 ts_tablet_manager.cc:1918] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:35:56.350622 8503 log.cc:1196] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/0d43c263b38b428a8798bb4317de798d
I20260926 11:35:56.350926 8503 ts_tablet_manager.cc:1939] T 0d43c263b38b428a8798bb4317de798d P fbd515b41912426eb996b9b50b03eda9: Deleting consensus metadata
I20260926 11:35:56.353878 7912 catalog_manager.cc:4841] TS fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531): tablet 0d43c263b38b428a8798bb4317de798d (table test-table [id=dc5c34a782eb43d2b42222b1e55bd911]) successfully deleted
I20260926 11:35:56.358057 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 190c48d8c15845d8a081b7a76ff3b308 (DEFAULT_TABLE table=test-table [id=408c44f4d8cf483bb3c1ca3cb327eb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:56.358074 8202 tablet_service.cc:1467] Processing CreateTablet for tablet 190c48d8c15845d8a081b7a76ff3b308 (DEFAULT_TABLE table=test-table [id=408c44f4d8cf483bb3c1ca3cb327eb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:56.358475 8202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190c48d8c15845d8a081b7a76ff3b308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:56.358475 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190c48d8c15845d8a081b7a76ff3b308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:56.360397 8343 tablet_service.cc:1467] Processing CreateTablet for tablet 190c48d8c15845d8a081b7a76ff3b308 (DEFAULT_TABLE table=test-table [id=408c44f4d8cf483bb3c1ca3cb327eb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:35:56.360790 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190c48d8c15845d8a081b7a76ff3b308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:56.361660 8437 tablet_bootstrap.cc:492] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9: Bootstrap starting.
I20260926 11:35:56.361660 8439 tablet_bootstrap.cc:492] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: Bootstrap starting.
I20260926 11:35:56.363029 8437 tablet_bootstrap.cc:654] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:56.363950 8438 tablet_bootstrap.cc:492] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: Bootstrap starting.
I20260926 11:35:56.364058 8439 tablet_bootstrap.cc:654] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:56.365015 8438 tablet_bootstrap.cc:654] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:56.365553 8437 tablet_bootstrap.cc:492] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9: No bootstrap required, opened a new log
I20260926 11:35:56.365666 8437 ts_tablet_manager.cc:1397] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 11:35:56.365723 8439 tablet_bootstrap.cc:492] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: No bootstrap required, opened a new log
I20260926 11:35:56.365828 8439 ts_tablet_manager.cc:1397] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 11:35:56.366279 8439 raft_consensus.cc:348] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.366279 8437 raft_consensus.cc:348] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.366436 8437 raft_consensus.cc:374] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:56.366436 8439 raft_consensus.cc:374] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:56.366482 8437 raft_consensus.cc:729] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd515b41912426eb996b9b50b03eda9, State: Initialized, Role: FOLLOWER
I20260926 11:35:56.366482 8439 raft_consensus.cc:729] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95364e6bf233428d9b601610ff2d4f16, State: Initialized, Role: FOLLOWER
I20260926 11:35:56.366624 8437 consensus_queue.cc:260] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.366624 8439 consensus_queue.cc:260] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.366914 8439 ts_tablet_manager.cc:1428] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:56.366914 8437 ts_tablet_manager.cc:1428] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 11:35:56.368360 8438 tablet_bootstrap.cc:492] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: No bootstrap required, opened a new log
I20260926 11:35:56.368471 8438 ts_tablet_manager.cc:1397] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 11:35:56.368927 8438 raft_consensus.cc:348] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.369059 8438 raft_consensus.cc:374] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:56.369091 8438 raft_consensus.cc:729] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f5db2578014a3a8637753eed3bb5dc, State: Initialized, Role: FOLLOWER
I20260926 11:35:56.369236 8438 consensus_queue.cc:260] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.369508 8438 ts_tablet_manager.cc:1428] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:35:56.515179 8530 raft_consensus.cc:482] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:35:56.515439 8530 raft_consensus.cc:504] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.516213 8530 leader_election.cc:288] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697), e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507)
I20260926 11:35:56.517058 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190c48d8c15845d8a081b7a76ff3b308" candidate_uuid: "fbd515b41912426eb996b9b50b03eda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95364e6bf233428d9b601610ff2d4f16" is_pre_election: true
I20260926 11:35:56.517095 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190c48d8c15845d8a081b7a76ff3b308" candidate_uuid: "fbd515b41912426eb996b9b50b03eda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f5db2578014a3a8637753eed3bb5dc" is_pre_election: true
I20260926 11:35:56.517295 8223 raft_consensus.cc:2454] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd515b41912426eb996b9b50b03eda9 in term 0.
I20260926 11:35:56.517382 8363 raft_consensus.cc:2454] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd515b41912426eb996b9b50b03eda9 in term 0.
I20260926 11:35:56.517751 8015 leader_election.cc:302] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95364e6bf233428d9b601610ff2d4f16, fbd515b41912426eb996b9b50b03eda9; no voters:
I20260926 11:35:56.518131 8530 raft_consensus.cc:2781] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:35:56.518219 8530 raft_consensus.cc:482] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:35:56.518252 8530 raft_consensus.cc:3037] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:56.519763 8530 raft_consensus.cc:504] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.520211 8530 leader_election.cc:288] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [CANDIDATE]: Term 1 election: Requested vote from peers 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697), e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507)
I20260926 11:35:56.520843 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190c48d8c15845d8a081b7a76ff3b308" candidate_uuid: "fbd515b41912426eb996b9b50b03eda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95364e6bf233428d9b601610ff2d4f16"
I20260926 11:35:56.520982 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190c48d8c15845d8a081b7a76ff3b308" candidate_uuid: "fbd515b41912426eb996b9b50b03eda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f5db2578014a3a8637753eed3bb5dc"
I20260926 11:35:56.521065 8223 raft_consensus.cc:3037] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:56.521193 8363 raft_consensus.cc:3037] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:56.522423 8363 raft_consensus.cc:2454] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd515b41912426eb996b9b50b03eda9 in term 1.
I20260926 11:35:56.522621 8223 raft_consensus.cc:2454] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd515b41912426eb996b9b50b03eda9 in term 1.
I20260926 11:35:56.522840 8015 leader_election.cc:302] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8f5db2578014a3a8637753eed3bb5dc, fbd515b41912426eb996b9b50b03eda9; no voters:
I20260926 11:35:56.523043 8530 raft_consensus.cc:2781] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:56.523241 8530 raft_consensus.cc:686] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 1 LEADER]: Becoming Leader. State: Replica: fbd515b41912426eb996b9b50b03eda9, State: Running, Role: LEADER
I20260926 11:35:56.523766 8530 consensus_queue.cc:237] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } }
I20260926 11:35:56.526325 7925 catalog_manager.cc:5495] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbd515b41912426eb996b9b50b03eda9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fbd515b41912426eb996b9b50b03eda9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbd515b41912426eb996b9b50b03eda9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 } health_report { overall_health: UNKNOWN } } }
I20260926 11:35:56.592516 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.587992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52231 (local address 127.0.98.2:38697)
0926 11:35:56.588204 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:56.588208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:56.588601 (+ 393us) server_negotiation.cc:407] Connection header received
0926 11:35:56.588675 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:56.588677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:56.588743 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:56.588843 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:56.589179 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.589933 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.591059 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.591302 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.591624 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.591991 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:56.592084 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:56.592155 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:56.592278 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 11:35:56.592337 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 11:35:56.592521 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.587812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38697 (local address 127.0.98.1:52231)
0926 11:35:56.588336 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:56.588352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:56.588499 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:56.588888 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:56.588891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:56.588900 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:56.589041 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:56.589052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.590087 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.590091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:56.590894 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:56.590905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.591629 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.591631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:56.591706 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.591829 (+ 123us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:56.592186 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:56.592191 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:35:56.592253 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 11:35:56.592359 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":168,"threads_started":1}
I20260926 11:35:56.601022 8363 raft_consensus.cc:1261] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Refusing update from remote peer fbd515b41912426eb996b9b50b03eda9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:56.601172 8223 raft_consensus.cc:1261] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Refusing update from remote peer fbd515b41912426eb996b9b50b03eda9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:35:56.601536 8530 consensus_queue.cc:1035] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f5db2578014a3a8637753eed3bb5dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:56.601748 8445 consensus_queue.cc:1035] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95364e6bf233428d9b601610ff2d4f16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:35:56.627635 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.622869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48069 (local address 127.0.98.1:38531)
0926 11:35:56.623006 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:56.623011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:56.623033 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:35:56.623236 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:56.623242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:56.623304 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:56.623399 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:35:56.623844 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.624392 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.625730 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.626039 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.626178 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.626810 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:35:56.626929 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:35:56.627031 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:56.627234 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 11:35:56.627314 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:35:56.627636 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.622568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38531 (local address 127.0.98.2:48069)
0926 11:35:56.622927 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:56.622967 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:56.623095 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:56.623476 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:56.623479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:56.623489 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:35:56.623714 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:56.623721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.624544 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.624547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:56.625518 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:56.625536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:56.626391 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:56.626395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:56.626587 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:56.626640 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:35:56.627064 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:35:56.627074 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 11:35:56.627153 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 11:35:56.627277 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":164,"threads_started":1}
I20260926 11:35:56.662921 7922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:56.663276 7922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:35:56.666754 7922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2eee0939d8604cffbec6c72e7145b837: Sending DeleteTablet for 3 replicas of tablet 190c48d8c15845d8a081b7a76ff3b308
I20260926 11:35:56.667832 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet 190c48d8c15845d8a081b7a76ff3b308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45776
I20260926 11:35:56.667865 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet 190c48d8c15845d8a081b7a76ff3b308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55730
I20260926 11:35:56.668105 8503 tablet_replica.cc:330] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9: stopping tablet replica
I20260926 11:35:56.668139 8504 tablet_replica.cc:330] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: stopping tablet replica
I20260926 11:35:56.668205 8202 tablet_service.cc:1514] Processing DeleteTablet for tablet 190c48d8c15845d8a081b7a76ff3b308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:35:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33664
I20260926 11:35:56.668279 8503 raft_consensus.cc:2229] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:35:56.668292 8504 raft_consensus.cc:2229] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:56.668445 8504 raft_consensus.cc:2258] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:56.668448 8505 tablet_replica.cc:330] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: stopping tablet replica
I20260926 11:35:56.668561 8503 raft_consensus.cc:2258] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:56.668613 8505 raft_consensus.cc:2229] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:35:56.668772 8505 raft_consensus.cc:2258] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:35:56.669426 8504 ts_tablet_manager.cc:1905] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:56.669543 8503 ts_tablet_manager.cc:1905] T 190c48d8c15845d8a081b7a76ff3b308 P fbd515b41912426eb996b9b50b03eda9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:56.669667 8505 ts_tablet_manager.cc:1905] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:35:56.670745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 7998
I20260926 11:35:56.673308 8505 ts_tablet_manager.cc:1918] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:56.673429 8505 log.cc:1196] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/190c48d8c15845d8a081b7a76ff3b308
I20260926 11:35:56.673749 8505 ts_tablet_manager.cc:1939] T 190c48d8c15845d8a081b7a76ff3b308 P 95364e6bf233428d9b601610ff2d4f16: Deleting consensus metadata
I20260926 11:35:56.674450 8504 ts_tablet_manager.cc:1918] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:35:56.674607 8504 log.cc:1196] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/190c48d8c15845d8a081b7a76ff3b308
I20260926 11:35:56.674917 8504 ts_tablet_manager.cc:1939] T 190c48d8c15845d8a081b7a76ff3b308 P e8f5db2578014a3a8637753eed3bb5dc: Deleting consensus metadata
I20260926 11:35:56.674880 7911 catalog_manager.cc:4841] TS 95364e6bf233428d9b601610ff2d4f16 (127.0.98.2:38697): tablet 190c48d8c15845d8a081b7a76ff3b308 (table test-table [id=408c44f4d8cf483bb3c1ca3cb327eb73]) successfully deleted
I20260926 11:35:56.675808 7911 catalog_manager.cc:4841] TS e8f5db2578014a3a8637753eed3bb5dc (127.0.98.3:36507): tablet 190c48d8c15845d8a081b7a76ff3b308 (table test-table [id=408c44f4d8cf483bb3c1ca3cb327eb73]) successfully deleted
W20260926 11:35:56.681780 7912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38531 (error 108)
I20260926 11:35:56.681988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 8138
I20260926 11:35:56.684782 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:56.683491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38531 (local address 127.0.0.1:45644)
0926 11:35:56.684458 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:56.684572 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":99,"threads_started":1}
W20260926 11:35:56.685024 7912 catalog_manager.cc:4584] TS fbd515b41912426eb996b9b50b03eda9 (127.0.98.1:38531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 190c48d8c15845d8a081b7a76ff3b308: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38531: connect: Connection refused (error 111)
I20260926 11:35:56.690966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 8278
I20260926 11:35:56.699285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 7894
2026-09-26T11:35:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3885 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:56 dist-test-slave-4pw5 krb5kdc[8550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:56 dist-test-slave-4pw5 krb5kdc[8550](info): set up 2 sockets
Sep 26 11:35:56 dist-test-slave-4pw5 krb5kdc[8550](info): commencing operation
krb5kdc: starting...
W20260926 11:35:57.788882 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:57 dist-test-slave-4pw5 krb5kdc[8550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:57.932317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34623
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:58.060139 8563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:58.060473 8563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:58.060523 8563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:58.065213 8563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:58.065318 8563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:58.065343 8563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:58.065367 8563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:58.065387 8563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:58.071206 8563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:58.072474 8563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:58.073585 8563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:35:58.083331 8563 server_base.cc:1034] running on GCE node
W20260926 11:35:58.083284 8568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:58.083267 8569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:58.083283 8571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:58.084318 8563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:58.085579 8563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:58.086809 8563 hybrid_clock.cc:648] HybridClock initialized: now 1790422558086795 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T11:35:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1388 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:35:58 dist-test-slave-4pw5 krb5kdc[8576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:35:58 dist-test-slave-4pw5 krb5kdc[8576](info): set up 2 sockets
Sep 26 11:35:58 dist-test-slave-4pw5 krb5kdc[8576](info): commencing operation
krb5kdc: starting...
W20260926 11:35:59.225028 392 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:35:59 dist-test-slave-4pw5 krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:35:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:35:59.361094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36101
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:59.474503 8589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:59.474911 8589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:59.474998 8589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:59.479314 8589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:35:59.479434 8589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:59.479472 8589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:59.479509 8589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:35:59.479562 8589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:35:59.486104 8589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:59.487329 8589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:59.488489 8589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:59.497896 8595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:59.497954 8594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:59.498062 8589 server_base.cc:1034] running on GCE node
W20260926 11:35:59.497896 8597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:59.498978 8589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:59.500322 8589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:59.501547 8589 hybrid_clock.cc:648] HybridClock initialized: now 1790422559501540 us; error 41 us; skew 500 ppm
Sep 26 11:35:59 dist-test-slave-4pw5 krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:59.504706 8589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:35:59.505825 8589 webserver.cc:460] Webserver started at http://127.0.98.62:41021/ using document root <none> and password file <none>
I20260926 11:35:59.506488 8589 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:59.506588 8589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:59.506839 8589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:59.509235 8589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "efcaba1f45b746ee81611c33d1bcf76a"
format_stamp: "Formatted at 2026-09-26 11:35:59 on dist-test-slave-4pw5"
server_key: "cbca31dc2f896fbcd0c411829ccb86e3"
server_key_iv: "82392f2099f934be14be333dd5526ce0"
server_key_version: "encryptionkey@0"
I20260926 11:35:59.509879 8589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "efcaba1f45b746ee81611c33d1bcf76a"
format_stamp: "Formatted at 2026-09-26 11:35:59 on dist-test-slave-4pw5"
server_key: "cbca31dc2f896fbcd0c411829ccb86e3"
server_key_iv: "82392f2099f934be14be333dd5526ce0"
server_key_version: "encryptionkey@0"
I20260926 11:35:59.514448 8589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:35:59.517362 8604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:59.519160 8589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:59.519366 8589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "efcaba1f45b746ee81611c33d1bcf76a"
format_stamp: "Formatted at 2026-09-26 11:35:59 on dist-test-slave-4pw5"
server_key: "cbca31dc2f896fbcd0c411829ccb86e3"
server_key_iv: "82392f2099f934be14be333dd5526ce0"
server_key_version: "encryptionkey@0"
I20260926 11:35:59.519515 8589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:59.541729 8589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:59.545305 8589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:59.545637 8589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:59.553995 8589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36101
I20260926 11:35:59.554026 8656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36101 every 8 connection(s)
I20260926 11:35:59.555147 8589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:35:59.558270 8657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:35:59.558854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 8589
I20260926 11:35:59.558997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:35:59.559310 392 external_mini_cluster.cc:1428] Setting key e1e01bf605a34596faee3ba8b6e1acc9
I20260926 11:35:59.565655 8657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: Bootstrap starting.
I20260926 11:35:59.568581 8657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: Neither blocks nor log segments found. Creating new log.
I20260926 11:35:59.569321 8657 log.cc:824] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:35:59.571511 8657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: No bootstrap required, opened a new log
Sep 26 11:35:59 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:35:59.576089 8657 raft_consensus.cc:348] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260926 11:35:59.576365 8657 raft_consensus.cc:374] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:35:59.576457 8657 raft_consensus.cc:729] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcaba1f45b746ee81611c33d1bcf76a, State: Initialized, Role: FOLLOWER
I20260926 11:35:59.576924 8657 consensus_queue.cc:260] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260926 11:35:59.577051 8657 raft_consensus.cc:388] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:35:59.577142 8657 raft_consensus.cc:482] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:35:59.577246 8657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:35:59.578231 8657 raft_consensus.cc:504] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260926 11:35:59.578621 8657 leader_election.cc:302] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efcaba1f45b746ee81611c33d1bcf76a; no voters:
I20260926 11:35:59.578990 8657 leader_election.cc:288] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:35:59.579074 8662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:35:59.579365 8662 raft_consensus.cc:686] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 1 LEADER]: Becoming Leader. State: Replica: efcaba1f45b746ee81611c33d1bcf76a, State: Running, Role: LEADER
I20260926 11:35:59.579759 8662 consensus_queue.cc:237] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260926 11:35:59.580535 8657 sys_catalog.cc:564] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:35:59.580771 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:59.560882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.0.98.62:36101)
0926 11:35:59.561349 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:59.561371 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:59.561426 (+ 55us) server_negotiation.cc:407] Connection header received
0926 11:35:59.562284 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:59.562308 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:59.562755 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:59.563154 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:59.564251 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:59.565739 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:59.567036 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:59.567636 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:59.567788 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:59.575052 (+ 7264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:59.575058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:59.575074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:59.575113 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:59.577477 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:59.577986 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:59.577993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:59.577999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:59.578131 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:59.578447 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:59.578453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:59.578458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:59.578969 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:59.579191 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:59.579439 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 11:35:59.579945 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":126,"threads_started":1}
I20260926 11:35:59.581053 8662 sys_catalog.cc:455] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efcaba1f45b746ee81611c33d1bcf76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } } }
I20260926 11:35:59.581197 8662 sys_catalog.cc:458] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:59.581459 8662 sys_catalog.cc:455] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: SysCatalogTable state changed. Reason: New leader efcaba1f45b746ee81611c33d1bcf76a. Latest consensus state: current_term: 1 leader_uuid: "efcaba1f45b746ee81611c33d1bcf76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } } }
I20260926 11:35:59.581565 8662 sys_catalog.cc:458] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: This master's current role is: LEADER
I20260926 11:35:59.583274 8668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:35:59.586140 8668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:35:59.593029 8668 catalog_manager.cc:1347] Generated new cluster ID: 862eea10fa884698bfa2337975d9a238
I20260926 11:35:59.593175 8668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:35:59.611445 8668 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:35:59.612522 8668 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:35:59.619143 8668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: Generated new TSK 0
I20260926 11:35:59.620018 8668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:35:59.663782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:35:59.791633 8683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:35:59.791954 8683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:35:59.792002 8683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:35:59.796473 8683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:35:59.796561 8683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:35:59.796625 8683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:35:59.802850 8683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:35:59.804124 8683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:35:59.805187 8683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:35:59.815730 8689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:59.815798 8691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:35:59.815846 8688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:35:59.815956 8683 server_base.cc:1034] running on GCE node
I20260926 11:35:59.816461 8683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:35:59.817164 8683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:35:59.818392 8683 hybrid_clock.cc:648] HybridClock initialized: now 1790422559818344 us; error 59 us; skew 500 ppm
Sep 26 11:35:59 dist-test-slave-4pw5 krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:35:59.821547 8683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:35:59.822643 8683 webserver.cc:460] Webserver started at http://127.0.98.1:46015/ using document root <none> and password file <none>
I20260926 11:35:59.823328 8683 fs_manager.cc:359] Metadata directory not provided
I20260926 11:35:59.823447 8683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:35:59.823738 8683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:35:59.825989 8683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "8fe432c188a24d9293ffeb8d1d70ac85"
format_stamp: "Formatted at 2026-09-26 11:35:59 on dist-test-slave-4pw5"
server_key: "bc73263135d318db81f15ad3ba13e114"
server_key_iv: "a2c0215654e3b49d7194e26290642628"
server_key_version: "encryptionkey@0"
I20260926 11:35:59.826735 8683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8fe432c188a24d9293ffeb8d1d70ac85"
format_stamp: "Formatted at 2026-09-26 11:35:59 on dist-test-slave-4pw5"
server_key: "bc73263135d318db81f15ad3ba13e114"
server_key_iv: "a2c0215654e3b49d7194e26290642628"
server_key_version: "encryptionkey@0"
I20260926 11:35:59.831167 8683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:35:59.833981 8698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:59.835501 8683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:35:59.835711 8683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "8fe432c188a24d9293ffeb8d1d70ac85"
format_stamp: "Formatted at 2026-09-26 11:35:59 on dist-test-slave-4pw5"
server_key: "bc73263135d318db81f15ad3ba13e114"
server_key_iv: "a2c0215654e3b49d7194e26290642628"
server_key_version: "encryptionkey@0"
I20260926 11:35:59.835830 8683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:35:59.869390 8683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:35:59.873107 8683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:35:59.873274 8683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:35:59.873831 8683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:35:59.874647 8683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:35:59.874696 8683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:59.874737 8683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:35:59.874753 8683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:35:59.885011 8683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35035
I20260926 11:35:59.885032 8811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35035 every 8 connection(s)
I20260926 11:35:59.885984 8683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:35:59.892020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 8683
I20260926 11:35:59.892284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:35:59.892608 392 external_mini_cluster.cc:1428] Setting key 96590c1b1ff932f1abdb70f99039cb3e
Sep 26 11:35:59 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:35:59.907882 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:59.888172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58335 (local address 127.0.98.62:36101)
0926 11:35:59.888317 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:35:59.888321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:35:59.889196 (+ 875us) server_negotiation.cc:407] Connection header received
0926 11:35:59.890614 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:35:59.890617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:35:59.890671 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:35:59.890738 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:35:59.892192 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:59.892934 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:59.894708 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:35:59.895074 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:35:59.895191 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:59.901628 (+ 6437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:35:59.901659 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:35:59.901666 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:35:59.901713 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:35:59.904132 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:59.905516 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:59.905521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:59.905525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:59.905641 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:35:59.906289 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:35:59.906293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:35:59.906295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:35:59.906586 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:35:59.906723 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 11:35:59.907488 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 11:35:59.907696 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:35:59.909469 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:35:59.888483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:58335)
0926 11:35:59.888985 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 11:35:59.889047 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 11:35:59.890352 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:35:59.891027 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:35:59.891034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:35:59.891434 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:35:59.892004 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:59.892019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:59.893075 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:59.893080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:59.894352 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:35:59.894364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:35:59.895408 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:35:59.895430 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:35:59.895885 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:35:59.898621 (+ 2736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:35:59.898656 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:35:59.901304 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:35:59.904378 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:59.904391 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:59.904419 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:59.905295 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:59.905780 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:35:59.905785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:35:59.905787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:35:59.906166 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:35:59.906717 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:35:59.906726 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:35:59.907136 (+ 410us) client_negotiation.cc:769] Sending connection context
0926 11:35:59.907568 (+ 432us) client_negotiation.cc:240] Negotiation successful
0926 11:35:59.908080 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":126,"threads_started":1}
I20260926 11:35:59.910965 8812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36101
I20260926 11:35:59.911234 8812 heartbeater.cc:464] Registering TS with master...
I20260926 11:35:59.912088 8812 heartbeater.cc:511] Master 127.0.98.62:36101 requested a full tablet report, sending...
I20260926 11:35:59.914026 8621 ts_manager.cc:194] Registered new tserver with Master: 8fe432c188a24d9293ffeb8d1d70ac85 (127.0.98.1:35035)
I20260926 11:35:59.915834 8621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:35:59.928869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:36:00.055658 8817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:00.055972 8817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:00.056044 8817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:00.060345 8817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:00.060421 8817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:00.060482 8817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:36:00.066108 8817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:00.067291 8817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:00.068496 8817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:00.079067 8825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:00.079069 8822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:00.079093 8823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:00.079497 8817 server_base.cc:1034] running on GCE node
I20260926 11:36:00.080179 8817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:00.081010 8817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:00.082216 8817 hybrid_clock.cc:648] HybridClock initialized: now 1790422560082193 us; error 39 us; skew 500 ppm
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:00.085394 8817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:36:00.086356 8817 webserver.cc:460] Webserver started at http://127.0.98.2:45905/ using document root <none> and password file <none>
I20260926 11:36:00.086985 8817 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:00.087049 8817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:00.087225 8817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:00.089385 8817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "a62df30c32d84956a33770f84a3fe9c8"
format_stamp: "Formatted at 2026-09-26 11:36:00 on dist-test-slave-4pw5"
server_key: "e7cca0ded9e78729d2842f9ebc137fee"
server_key_iv: "bfbe84038547021e15c41283b617b8c4"
server_key_version: "encryptionkey@0"
I20260926 11:36:00.089905 8817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a62df30c32d84956a33770f84a3fe9c8"
format_stamp: "Formatted at 2026-09-26 11:36:00 on dist-test-slave-4pw5"
server_key: "e7cca0ded9e78729d2842f9ebc137fee"
server_key_iv: "bfbe84038547021e15c41283b617b8c4"
server_key_version: "encryptionkey@0"
I20260926 11:36:00.093832 8817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 11:36:00.096606 8832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.097921 8817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:36:00.098063 8817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "a62df30c32d84956a33770f84a3fe9c8"
format_stamp: "Formatted at 2026-09-26 11:36:00 on dist-test-slave-4pw5"
server_key: "e7cca0ded9e78729d2842f9ebc137fee"
server_key_iv: "bfbe84038547021e15c41283b617b8c4"
server_key_version: "encryptionkey@0"
I20260926 11:36:00.098198 8817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:00.109927 8817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:00.113745 8817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:00.113977 8817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:00.114593 8817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:00.115485 8817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:00.115535 8817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.115653 8817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:00.115674 8817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.125777 8817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43877
I20260926 11:36:00.125890 8945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43877 every 8 connection(s)
I20260926 11:36:00.126787 8817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:36:00.127507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 8817
I20260926 11:36:00.127655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:36:00.127969 392 external_mini_cluster.cc:1428] Setting key cde68af4f3cdad03f8ae05b4963955c4
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:36:00.145829 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:00.129022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46371 (local address 127.0.98.62:36101)
0926 11:36:00.129170 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:00.129177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:00.130409 (+ 1232us) server_negotiation.cc:407] Connection header received
0926 11:36:00.132547 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:00.132553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:00.132638 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:00.132767 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:00.134166 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.135267 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.136678 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.137070 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.137208 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.140168 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:00.140171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:00.140174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:36:00.140210 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:00.141909 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:00.142656 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:00.142661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:00.142663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:00.142775 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:00.143205 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:00.143211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:00.143215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:00.143515 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:00.143658 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:00.145493 (+ 1835us) server_negotiation.cc:299] Negotiation successful
0926 11:36:00.145648 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:36:00.146724 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:00.129570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:46371)
0926 11:36:00.130205 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:00.130280 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:00.132232 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:00.132983 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:00.132992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:00.133394 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:00.133988 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.134004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.135471 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.135474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.136532 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.136544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.137235 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.137237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.137384 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.138124 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:00.138141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:00.139974 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:00.142069 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:00.142075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:00.142091 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:00.142528 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:00.142909 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:00.142912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:00.142913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:00.143080 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:00.143695 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:00.143702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:00.144001 (+ 299us) client_negotiation.cc:769] Sending connection context
0926 11:36:00.145408 (+ 1407us) client_negotiation.cc:240] Negotiation successful
0926 11:36:00.145799 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":135,"threads_started":1}
I20260926 11:36:00.148128 8946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36101
I20260926 11:36:00.148458 8946 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:00.149195 8946 heartbeater.cc:511] Master 127.0.98.62:36101 requested a full tablet report, sending...
I20260926 11:36:00.150823 8621 ts_manager.cc:194] Registered new tserver with Master: a62df30c32d84956a33770f84a3fe9c8 (127.0.98.2:43877)
I20260926 11:36:00.151767 8621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:36:00.164930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:36:00.281394 8951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:00.281886 8951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:00.281939 8951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:00.286679 8951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:00.286759 8951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:00.286823 8951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:36:00.292558 8951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:00.293687 8951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:00.294832 8951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:00.305640 8959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:00.305624 8956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:00.305603 8957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:00.305794 8951 server_base.cc:1034] running on GCE node
I20260926 11:36:00.306466 8951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:00.307214 8951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:00.308393 8951 hybrid_clock.cc:648] HybridClock initialized: now 1790422560308359 us; error 52 us; skew 500 ppm
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:00.311910 8951 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:36:00.312983 8951 webserver.cc:460] Webserver started at http://127.0.98.3:40763/ using document root <none> and password file <none>
I20260926 11:36:00.313594 8951 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:00.313660 8951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:00.313845 8951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:00.316304 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "c19648c0d100409980b9e1fc3e303a30"
format_stamp: "Formatted at 2026-09-26 11:36:00 on dist-test-slave-4pw5"
server_key: "943c404f185ae24a5b108c440b723832"
server_key_iv: "967f0658e70ccf058805c1c27bd9dd58"
server_key_version: "encryptionkey@0"
I20260926 11:36:00.316903 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c19648c0d100409980b9e1fc3e303a30"
format_stamp: "Formatted at 2026-09-26 11:36:00 on dist-test-slave-4pw5"
server_key: "943c404f185ae24a5b108c440b723832"
server_key_iv: "967f0658e70ccf058805c1c27bd9dd58"
server_key_version: "encryptionkey@0"
I20260926 11:36:00.321308 8951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:36:00.324728 8966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.326279 8951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:00.326453 8951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "c19648c0d100409980b9e1fc3e303a30"
format_stamp: "Formatted at 2026-09-26 11:36:00 on dist-test-slave-4pw5"
server_key: "943c404f185ae24a5b108c440b723832"
server_key_iv: "967f0658e70ccf058805c1c27bd9dd58"
server_key_version: "encryptionkey@0"
I20260926 11:36:00.326601 8951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:00.359349 8951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:00.362944 8951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:00.363171 8951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:00.363854 8951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:00.364755 8951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:00.364804 8951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.364845 8951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:00.364859 8951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.373621 8951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43535
I20260926 11:36:00.373646 9079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43535 every 8 connection(s)
I20260926 11:36:00.374603 8951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:36:00.382927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 8951
I20260926 11:36:00.383078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:36:00.383719 392 external_mini_cluster.cc:1428] Setting key be166a653270c860713aa66e21581218
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:36:00.389771 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:00.376630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55631 (local address 127.0.98.62:36101)
0926 11:36:00.376782 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:00.376785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:00.377663 (+ 878us) server_negotiation.cc:407] Connection header received
0926 11:36:00.379089 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:00.379093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:00.379159 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:00.379260 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:00.380519 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.381073 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.382322 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.382536 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.382643 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.385636 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:00.385640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:00.385643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:36:00.385670 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:00.387522 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:00.388140 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:00.388143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:00.388144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:00.388226 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:00.388614 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:00.388616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:00.388617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:00.388807 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:00.388886 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:00.389521 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 11:36:00.389644 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 11:36:00.390779 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:00.377045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:55631)
0926 11:36:00.377491 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:00.377533 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:00.378845 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:00.379434 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:00.379440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:00.379878 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:00.380360 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.380373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.381202 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.381208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.382190 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.382199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.382697 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.382701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.382884 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.383484 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:00.383500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:00.385463 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:00.387700 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:00.387706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:00.387721 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:00.388018 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:00.388320 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:00.388323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:00.388324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:00.388475 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:00.388913 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:00.388923 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:00.389205 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 11:36:00.389368 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 11:36:00.389707 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260926 11:36:00.392297 9080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36101
I20260926 11:36:00.392566 9080 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:00.393148 9080 heartbeater.cc:511] Master 127.0.98.62:36101 requested a full tablet report, sending...
I20260926 11:36:00.394357 8621 ts_manager.cc:194] Registered new tserver with Master: c19648c0d100409980b9e1fc3e303a30 (127.0.98.3:43535)
I20260926 11:36:00.394970 8621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55631
I20260926 11:36:00.398532 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 11:36:00.487627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 8683
I20260926 11:36:00.493919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 8817
I20260926 11:36:00.499792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 8951
I20260926 11:36:00.505541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 8589
I20260926 11:36:00.512248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36101
--webserver_interface=127.0.98.62
--webserver_port=41021
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:36:00.629437 9089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:00.629812 9089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:00.629901 9089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:00.634770 9089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:36:00.634923 9089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:00.634974 9089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:00.635017 9089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:36:00.635057 9089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:36:00.641126 9089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:00.642343 9089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:00.643433 9089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:36:00.652899 9089 server_base.cc:1034] running on GCE node
W20260926 11:36:00.652853 9094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:00.652853 9095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:00.652853 9097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:00.654072 9089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:00.655416 9089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:00.656662 9089 hybrid_clock.cc:648] HybridClock initialized: now 1790422560656645 us; error 37 us; skew 500 ppm
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:00.659861 9089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:36:00.660887 9089 webserver.cc:460] Webserver started at http://127.0.98.62:41021/ using document root <none> and password file <none>
I20260926 11:36:00.661454 9089 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:00.661530 9089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:00.665800 9089 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:36:00.668793 9104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.670462 9089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:00.670617 9089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "efcaba1f45b746ee81611c33d1bcf76a"
format_stamp: "Formatted at 2026-09-26 11:35:59 on dist-test-slave-4pw5"
server_key: "cbca31dc2f896fbcd0c411829ccb86e3"
server_key_iv: "82392f2099f934be14be333dd5526ce0"
server_key_version: "encryptionkey@0"
I20260926 11:36:00.671270 9089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:00.704551 9089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:00.708319 9089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:00.708498 9089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:00.716351 9089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36101
I20260926 11:36:00.716384 9156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36101 every 8 connection(s)
I20260926 11:36:00.717501 9089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:36:00.720026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 9089
I20260926 11:36:00.720552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35035
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36101
--webserver_port=46015
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:36:00.722862 9157 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:36:00.724918 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: Bootstrap starting.
I20260926 11:36:00.742918 9157 log.cc:824] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:36:00.750252 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:36:00.750706 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: Bootstrap complete.
I20260926 11:36:00.755226 9157 raft_consensus.cc:348] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260926 11:36:00.755740 9157 raft_consensus.cc:729] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcaba1f45b746ee81611c33d1bcf76a, State: Initialized, Role: FOLLOWER
I20260926 11:36:00.756321 9157 consensus_queue.cc:260] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260926 11:36:00.756454 9157 raft_consensus.cc:388] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:36:00.756543 9157 raft_consensus.cc:482] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:36:00.756637 9157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:36:00.758872 9157 raft_consensus.cc:504] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260926 11:36:00.759260 9157 leader_election.cc:302] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efcaba1f45b746ee81611c33d1bcf76a; no voters:
I20260926 11:36:00.759647 9157 leader_election.cc:288] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 11:36:00.759779 9161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 2 FOLLOWER]: Leader election won for term 2
I20260926 11:36:00.760041 9161 raft_consensus.cc:686] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [term 2 LEADER]: Becoming Leader. State: Replica: efcaba1f45b746ee81611c33d1bcf76a, State: Running, Role: LEADER
I20260926 11:36:00.760442 9161 consensus_queue.cc:237] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260926 11:36:00.761003 9157 sys_catalog.cc:564] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:36:00.761927 9162 sys_catalog.cc:455] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "efcaba1f45b746ee81611c33d1bcf76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } } }
I20260926 11:36:00.762080 9162 sys_catalog.cc:458] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: This master's current role is: LEADER
I20260926 11:36:00.762454 9163 sys_catalog.cc:455] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: SysCatalogTable state changed. Reason: New leader efcaba1f45b746ee81611c33d1bcf76a. Latest consensus state: current_term: 2 leader_uuid: "efcaba1f45b746ee81611c33d1bcf76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcaba1f45b746ee81611c33d1bcf76a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } } }
I20260926 11:36:00.762588 9163 sys_catalog.cc:458] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a [sys.catalog]: This master's current role is: LEADER
I20260926 11:36:00.762952 9170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:36:00.766314 9170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:36:00.766966 9170 catalog_manager.cc:1259] Loaded cluster ID: 862eea10fa884698bfa2337975d9a238
I20260926 11:36:00.767025 9170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:36:00.769881 9170 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:36:00.770702 9170 catalog_manager.cc:5879] T 00000000000000000000000000000000 P efcaba1f45b746ee81611c33d1bcf76a: Loaded TSK: 0
I20260926 11:36:00.772049 9170 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:36:00.850283 9159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:00.850570 9159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:00.850615 9159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:00.855103 9159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:00.855207 9159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:00.855314 9159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:36:00.861202 9159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46015
--tserver_master_addrs=127.0.98.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:00.862311 9159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:00.863428 9159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:00.875306 9183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:00.875310 9184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:00.875633 9186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:00.875924 9159 server_base.cc:1034] running on GCE node
I20260926 11:36:00.876408 9159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:00.877204 9159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:00.878466 9159 hybrid_clock.cc:648] HybridClock initialized: now 1790422560878429 us; error 60 us; skew 500 ppm
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:00.881654 9159 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 11:36:00.882684 9159 webserver.cc:460] Webserver started at http://127.0.98.1:46015/ using document root <none> and password file <none>
I20260926 11:36:00.883291 9159 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:00.883402 9159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:00.887773 9159 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:00.890133 9193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.891763 9159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:00.891943 9159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "8fe432c188a24d9293ffeb8d1d70ac85"
format_stamp: "Formatted at 2026-09-26 11:35:59 on dist-test-slave-4pw5"
server_key: "bc73263135d318db81f15ad3ba13e114"
server_key_iv: "a2c0215654e3b49d7194e26290642628"
server_key_version: "encryptionkey@0"
I20260926 11:36:00.892573 9159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:00.918568 9159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:00.922224 9159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:00.922447 9159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:00.923094 9159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:00.923998 9159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:00.924077 9159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.924171 9159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:00.924224 9159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:00.934250 9159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35035
I20260926 11:36:00.934266 9306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35035 every 8 connection(s)
I20260926 11:36:00.935544 9159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:36:00.938788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 9159
I20260926 11:36:00.939293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43877
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36101
--webserver_port=45905
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:36:00.953078 9309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:00.937963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56129 (local address 127.0.98.62:36101)
0926 11:36:00.938600 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:00.938623 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:00.938884 (+ 261us) server_negotiation.cc:407] Connection header received
0926 11:36:00.940990 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:00.941003 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:00.941342 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:00.941708 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:00.943618 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.944646 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.946860 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.947462 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.947728 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.952208 (+ 4480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56129: BlockingRecv error: recv got EOF from 127.0.98.1:56129 (error 108)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":142,"threads_started":1}
I20260926 11:36:00.953088 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:00.938041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:56129)
0926 11:36:00.938694 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:00.938748 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:00.940427 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:00.941885 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:00.941898 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:00.942495 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:00.943257 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.943283 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.944820 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.944826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.946666 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.946685 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.947720 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.947728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.947991 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.948727 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:00.948760 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:00.951947 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":158,"threads_started":1}
W20260926 11:36:00.953773 9307 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36101 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36101: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:00.960153 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:00.954329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:46747)
0926 11:36:00.954466 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:00.954485 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:00.954585 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:00.954865 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:00.954868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:00.955064 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:00.955168 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.955175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.956116 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.956123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.956731 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.956739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.957238 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.957242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.957353 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.957911 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:00.957924 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:00.960000 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 11:36:00.960474 9309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:00.954429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46747 (local address 127.0.98.62:36101)
0926 11:36:00.954586 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:00.954591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:00.954608 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:36:00.954689 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:00.954692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:00.954748 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:00.954847 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:00.955301 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.955968 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.956863 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.957110 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.957230 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.960201 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46747: BlockingRecv error: recv got EOF from 127.0.98.1:46747 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:00 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:00.966791 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:00.960797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:36147)
0926 11:36:00.960924 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:00.960990 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:00.961080 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:00.961385 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:00.961389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:00.961559 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:00.961673 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.961681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.962520 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.962524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.963224 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:00.963234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.963814 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.963819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:00.963924 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.964448 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:00.964463 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:00.966630 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 11:36:00.966998 9309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:00.960940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36147 (local address 127.0.98.62:36101)
0926 11:36:00.961064 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:00.961071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:00.961090 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:36:00.961185 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:00.961188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:00.961237 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:00.961351 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:00.961825 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.962398 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.963351 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:00.963668 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:00.963805 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:00.966801 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36147: BlockingRecv error: recv got EOF from 127.0.98.1:36147 (error 108)
Metrics: {"server-negotiator.queue_time_us":50,"thread_start_us":131,"threads_started":1}
W20260926 11:36:00.967070 9307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:36:01.068912 9311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:01.069234 9311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:01.069281 9311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:01.074118 9311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:01.074226 9311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:01.074293 9311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:36:01.080708 9311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45905
--tserver_master_addrs=127.0.98.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:01.081943 9311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:01.083119 9311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:01.093665 9320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:01.093743 9317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:01.093665 9318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:01.094092 9311 server_base.cc:1034] running on GCE node
I20260926 11:36:01.094722 9311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:01.095444 9311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:01.096619 9311 hybrid_clock.cc:648] HybridClock initialized: now 1790422561096600 us; error 38 us; skew 500 ppm
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:01.099849 9311 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 11:36:01.100903 9311 webserver.cc:460] Webserver started at http://127.0.98.2:45905/ using document root <none> and password file <none>
I20260926 11:36:01.101583 9311 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:01.101681 9311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:01.106053 9311 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 11:36:01.108809 9327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:01.110273 9311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:01.110620 9311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "a62df30c32d84956a33770f84a3fe9c8"
format_stamp: "Formatted at 2026-09-26 11:36:00 on dist-test-slave-4pw5"
server_key: "e7cca0ded9e78729d2842f9ebc137fee"
server_key_iv: "bfbe84038547021e15c41283b617b8c4"
server_key_version: "encryptionkey@0"
I20260926 11:36:01.111279 9311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:01.129361 9311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:01.133662 9311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:01.133919 9311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:01.134627 9311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:01.135555 9311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:01.135699 9311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:01.135771 9311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:01.135820 9311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:01.146005 9311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43877
I20260926 11:36:01.146075 9440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43877 every 8 connection(s)
I20260926 11:36:01.147163 9311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:36:01.147538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 9311
I20260926 11:36:01.148084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43535
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36101
--webserver_port=40763
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:36:01.161350 9309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:01.149275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45645 (local address 127.0.98.62:36101)
0926 11:36:01.149414 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:01.149419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:01.150452 (+ 1033us) server_negotiation.cc:407] Connection header received
0926 11:36:01.152213 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:01.152219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:01.152302 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:01.152426 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:01.153882 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.154829 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.156254 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.156675 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.156821 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.161196 (+ 4375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45645: BlockingRecv error: recv got EOF from 127.0.98.2:45645 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:36:01.162045 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:01.149755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:45645)
0926 11:36:01.150273 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:01.150330 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:01.151919 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:01.152667 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:01.152675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:01.153076 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:01.153682 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.153700 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.154998 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.155002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.156106 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.156118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.156859 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.156862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.157020 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.157816 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:01.157833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:01.160992 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":156,"threads_started":1}
W20260926 11:36:01.162711 9441 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36101 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36101: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:01.169893 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:01.163278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:59095)
0926 11:36:01.163384 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:01.163400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:01.163494 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:01.163794 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:01.163796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:01.163958 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:01.164070 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.164075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.165358 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.165361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.165950 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.165960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.166654 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.166656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.166732 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.167392 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:01.167405 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:01.169770 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 11:36:01.170079 9309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:01.163395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59095 (local address 127.0.98.62:36101)
0926 11:36:01.163524 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:01.163529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:01.163544 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:36:01.163618 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:01.163621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:01.163671 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:01.163768 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:01.164202 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.165144 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.166102 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.166514 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.166655 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.169950 (+ 3295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59095: BlockingRecv error: recv got EOF from 127.0.98.2:59095 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:01.176766 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:01.170701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:53197)
0926 11:36:01.170914 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:01.170927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:01.170997 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:01.171265 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:01.171267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:01.171401 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:01.171534 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.171539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.172654 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.172657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.173200 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.173208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.173735 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.173738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.173799 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.174347 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:01.174359 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:01.176592 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131}
W20260926 11:36:01.176846 9309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:01.170821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53197 (local address 127.0.98.62:36101)
0926 11:36:01.170950 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:01.170955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:01.170970 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:36:01.171088 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:01.171091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:01.171143 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:01.171230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:01.171693 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.172496 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.173320 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.173616 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.173717 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.176730 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53197: BlockingRecv error: recv got EOF from 127.0.98.2:53197 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 11:36:01.177007 9441 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:36:01.279048 9443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:01.279356 9443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:01.279402 9443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:01.284027 9443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:01.284147 9443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:01.284229 9443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:36:01.290028 9443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40763
--tserver_master_addrs=127.0.98.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:01.291306 9443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:01.292569 9443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:01.303776 9449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:01.303764 9452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:01.303764 9450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:01.304188 9443 server_base.cc:1034] running on GCE node
I20260926 11:36:01.304687 9443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:01.305436 9443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:01.306664 9443 hybrid_clock.cc:648] HybridClock initialized: now 1790422561306629 us; error 48 us; skew 500 ppm
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:01.309787 9443 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 11:36:01.310778 9443 webserver.cc:460] Webserver started at http://127.0.98.3:40763/ using document root <none> and password file <none>
I20260926 11:36:01.311414 9443 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:01.311616 9443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:01.316377 9443 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:01.318857 9459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:01.320400 9443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:01.320602 9443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "c19648c0d100409980b9e1fc3e303a30"
format_stamp: "Formatted at 2026-09-26 11:36:00 on dist-test-slave-4pw5"
server_key: "943c404f185ae24a5b108c440b723832"
server_key_iv: "967f0658e70ccf058805c1c27bd9dd58"
server_key_version: "encryptionkey@0"
I20260926 11:36:01.321226 9443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:01.336746 9443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:01.340608 9443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:01.340849 9443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:01.341540 9443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:01.342549 9443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:01.342644 9443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:01.342724 9443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:01.342772 9443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:01.353536 9443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43535
I20260926 11:36:01.353560 9572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43535 every 8 connection(s)
I20260926 11:36:01.354904 9443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:36:01.355756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 9443
I20260926 11:36:01.366813 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:01.356604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.0.98.62:36101)
0926 11:36:01.356772 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:01.356777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:01.356951 (+ 174us) server_negotiation.cc:407] Connection header received
0926 11:36:01.357093 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:01.357097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:01.357149 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:01.357240 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:01.358158 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.358742 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.359649 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.359934 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.360056 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.361802 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:01.361809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:01.361845 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:36:01.361879 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:01.364231 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:01.364822 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:01.364827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:01.364832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:01.365025 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:01.365310 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:01.365314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:01.365315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:01.365829 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:01.366045 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:01.366344 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 11:36:01.366624 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":66}
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:36:01.369117 9312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:01.357521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46027 (local address 127.0.98.62:36101)
0926 11:36:01.357687 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:01.357700 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:01.358395 (+ 695us) server_negotiation.cc:407] Connection header received
0926 11:36:01.359836 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:01.359839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:01.359891 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:01.359986 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:01.361750 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.362579 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.364422 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.364857 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.364942 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.368975 (+ 4033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46027: BlockingRecv error: recv got EOF from 127.0.98.3:46027 (error 108)
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":73,"spinlock_wait_cycles":4352}
I20260926 11:36:01.369831 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:01.357726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:46027)
0926 11:36:01.358231 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:01.358285 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:01.359637 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:01.360233 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:01.360246 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:01.360755 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:01.361511 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.361533 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.362720 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.362725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.364295 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.364308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.365021 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.365026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.365235 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.365917 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:01.365938 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:01.368846 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":161,"threads_started":1}
W20260926 11:36:01.370399 9573 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36101 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36101: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:01.376562 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:01.370938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:53441)
0926 11:36:01.371100 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:01.371120 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:01.371193 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:01.371522 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:01.371524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:01.371753 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:01.371856 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.371862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.372643 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.372645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.373195 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.373202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.373674 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.373677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.373755 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.374239 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:01.374260 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:01.376395 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 11:36:01.376708 9312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:01.371007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53441 (local address 127.0.98.62:36101)
0926 11:36:01.371145 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:01.371149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:01.371173 (+ 24us) server_negotiation.cc:407] Connection header received
0926 11:36:01.371276 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:01.371278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:01.371326 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:01.371402 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:01.372001 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.372534 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.373317 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.373546 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.373637 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.376584 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53441: BlockingRecv error: recv got EOF from 127.0.98.3:53441 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:01.383253 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:01.377271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:56071)
0926 11:36:01.377392 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:01.377408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:01.377503 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:01.377877 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:01.377880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:01.378029 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:01.378114 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.378120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.379005 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.379008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.379657 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.379669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.380245 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.380248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.380313 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.380886 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:01.380903 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:01.383134 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 11:36:01.383474 9312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:01.377464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56071 (local address 127.0.98.62:36101)
0926 11:36:01.377621 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:01.377625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:01.377638 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:36:01.377691 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:01.377693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:01.377735 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:01.377822 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:01.378247 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.378812 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.379835 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.380131 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.380226 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.383316 (+ 3090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56071: BlockingRecv error: recv got EOF from 127.0.98.3:56071 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 11:36:01.383667 9573 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:01 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:01.975711 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:01.968070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:49925)
0926 11:36:01.968509 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:01.968533 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:01.968741 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:01.969227 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:01.969231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:01.969602 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:01.969745 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.969752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.970960 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.970963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.971640 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:01.971652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.972207 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.972211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:01.972292 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.972984 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:01.972997 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:01.975452 (+ 2455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":126,"threads_started":1}
W20260926 11:36:01.976104 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:01.968293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49925 (local address 127.0.98.62:36101)
0926 11:36:01.968747 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:01.968752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:01.968777 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:36:01.968866 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:01.968872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:01.968942 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:01.969082 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:01.969937 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.970767 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.971797 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:01.972044 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:01.972137 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:01.975895 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49925: BlockingRecv error: recv got EOF from 127.0.98.1:49925 (error 108)
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":176,"threads_started":1}
Sep 26 11:36:02 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:02 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:02.185088 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:02.177953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:60321)
0926 11:36:02.178242 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:02.178256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:02.178402 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:02.178963 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:02.178967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:02.179246 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:02.179396 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:02.179403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.180341 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.180345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:02.180946 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:02.180956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.181449 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.181453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:02.181571 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:02.182261 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:02.182274 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:02.184853 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
W20260926 11:36:02.185170 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:02.178014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60321 (local address 127.0.98.62:36101)
0926 11:36:02.178305 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:02.178315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:02.178337 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:36:02.178611 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:02.178617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:02.178692 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:02.178827 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:02.179539 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.180194 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.181077 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.181323 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.181441 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:02.185032 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60321: BlockingRecv error: recv got EOF from 127.0.98.2:60321 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 11:36:02 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:02 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:02.392645 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:02.384823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:45135)
0926 11:36:02.385159 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:02.385183 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:02.385361 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:02.385884 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:02.385897 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:02.386152 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:02.386324 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:02.386338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.387845 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.387848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:02.388558 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:02.388571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.389197 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.389200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:02.389286 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:02.389996 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:02.390012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:02.392439 (+ 2427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":119,"threads_started":1}
W20260926 11:36:02.392884 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:02.384959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45135 (local address 127.0.98.62:36101)
0926 11:36:02.385154 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:02.385159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:02.385285 (+ 126us) server_negotiation.cc:407] Connection header received
0926 11:36:02.385576 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:02.385583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:02.385674 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:02.385795 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:02.386682 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.387506 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.388744 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.389002 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.389111 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:02.392694 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45135: BlockingRecv error: recv got EOF from 127.0.98.3:45135 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 11:36:02 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:02 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:02.983716 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:02.977025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:49087)
0926 11:36:02.977392 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:02.977409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:02.977565 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:02.977951 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:02.977954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:02.978157 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:02.978268 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:02.978274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.979179 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.979182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:02.979887 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:02.979899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.980426 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.980429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:02.980502 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:02.981097 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:02.981108 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:02.983449 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
W20260926 11:36:02.983907 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:02.977124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49087 (local address 127.0.98.62:36101)
0926 11:36:02.977489 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:02.977494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:02.977514 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:36:02.977693 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:02.977696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:02.977753 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:02.977858 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:02.978409 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.979033 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.980031 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:02.980283 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:02.980377 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:02.983734 (+ 3357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49087: BlockingRecv error: recv got EOF from 127.0.98.1:49087 (error 108)
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":127,"threads_started":1}
Sep 26 11:36:03 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:03 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:03.193522 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:03.186370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:39865)
0926 11:36:03.186752 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:03.186767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:03.186902 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:03.187262 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:03.187265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:03.187497 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:03.187736 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:03.187744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.188580 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.188584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:03.189238 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:03.189249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.189804 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.189807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:03.189887 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:03.190649 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:03.190663 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:03.193270 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
W20260926 11:36:03.193630 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:03.186475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39865 (local address 127.0.98.62:36101)
0926 11:36:03.186684 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:03.186688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:03.186853 (+ 165us) server_negotiation.cc:407] Connection header received
0926 11:36:03.187016 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:03.187018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:03.187075 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:03.187161 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:03.187868 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.188433 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.189399 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.189641 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.189742 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:03.193489 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39865: BlockingRecv error: recv got EOF from 127.0.98.2:39865 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 26 11:36:03 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:03 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:03.401535 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:03.393823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:52795)
0926 11:36:03.394355 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:03.394377 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:03.394581 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:03.394921 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:03.394924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:03.395189 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:03.395307 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:03.395313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.396239 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.396246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:03.396984 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:03.396995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.397688 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.397696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:03.397923 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:03.398701 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:03.398719 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:03.401352 (+ 2633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":121,"threads_started":1}
W20260926 11:36:03.401752 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:03.394095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52795 (local address 127.0.98.62:36101)
0926 11:36:03.394355 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:03.394362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:03.394494 (+ 132us) server_negotiation.cc:407] Connection header received
0926 11:36:03.394712 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:03.394718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:03.394789 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:03.394889 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:03.395439 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.396106 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.397125 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.397460 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.397578 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:03.401572 (+ 3994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52795: BlockingRecv error: recv got EOF from 127.0.98.3:52795 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 26 11:36:03 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:03 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:03.990687 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:03.984842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:42619)
0926 11:36:03.985203 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:03.985239 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:03.985350 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:03.985685 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:03.985687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:03.985868 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:03.985972 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:03.985977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.986764 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.986767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:03.987353 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:03.987361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.987885 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.987888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:03.987960 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:03.988490 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:03.988502 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:03.990475 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":86,"threads_started":1}
W20260926 11:36:03.990808 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:03.984959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42619 (local address 127.0.98.62:36101)
0926 11:36:03.985244 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:03.985248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:03.985297 (+ 49us) server_negotiation.cc:407] Connection header received
0926 11:36:03.985468 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:03.985470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:03.985516 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:03.985625 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:03.986074 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.986653 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.987508 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:03.987779 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:03.987860 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:03.990663 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42619: BlockingRecv error: recv got EOF from 127.0.98.1:42619 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
Sep 26 11:36:04 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:04 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:04.200716 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:04.194570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:49221)
0926 11:36:04.194870 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:04.194885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:04.195010 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:04.195283 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:04.195286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:04.195516 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:04.195744 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:04.195751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.196611 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.196614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:04.197274 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:04.197285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.197844 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.197848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:04.197944 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:04.198469 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:04.198481 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:04.200561 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":114,"threads_started":1}
W20260926 11:36:04.200904 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:04.194657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49221 (local address 127.0.98.62:36101)
0926 11:36:04.194797 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:04.194801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:04.194967 (+ 166us) server_negotiation.cc:407] Connection header received
0926 11:36:04.195093 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:04.195096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:04.195138 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:04.195214 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:04.195857 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.196466 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.197411 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.197683 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.197768 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:04.200750 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49221: BlockingRecv error: recv got EOF from 127.0.98.2:49221 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 11:36:04 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:04 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:04.408403 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:04.402491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:42925)
0926 11:36:04.402774 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:04.402787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:04.402903 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:04.403295 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:04.403297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:04.403485 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:04.403633 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:04.403639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.404459 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.404462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:04.405047 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:04.405059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.405537 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.405540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:04.405598 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:04.406200 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:04.406213 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:04.408243 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
W20260926 11:36:04.408486 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:04.402597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42925 (local address 127.0.98.62:36101)
0926 11:36:04.402750 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:04.402753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:04.402885 (+ 132us) server_negotiation.cc:407] Connection header received
0926 11:36:04.403006 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:04.403009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:04.403074 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:04.403193 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:04.403749 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.404352 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.405172 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.405421 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.405524 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:04.408389 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42925: BlockingRecv error: recv got EOF from 127.0.98.3:42925 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 11:36:04 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:04 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:05.002281 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:04.991808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:57193)
0926 11:36:04.992104 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:04.992125 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:04.992253 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:04.992500 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:04.992502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:04.992709 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:04.992802 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:04.992807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.993663 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.993666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:04.994219 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:04.994227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.994735 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.994738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:04.994803 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:04.999986 (+ 5183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:05.000001 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:05.002120 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
W20260926 11:36:05.002682 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:04.991981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57193 (local address 127.0.98.62:36101)
0926 11:36:04.992274 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:04.992278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:04.992293 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:36:04.992337 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:04.992339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:04.992380 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:04.992467 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:04.992950 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.993547 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.994343 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:04.994614 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:04.994706 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:05.002358 (+ 7652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57193: BlockingRecv error: recv got EOF from 127.0.98.1:57193 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
Sep 26 11:36:05 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:05 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:05.209265 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:05.201746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:37581)
0926 11:36:05.202075 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:05.202089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:05.202191 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:05.202943 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:05.202947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:05.203265 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:05.203512 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:05.203522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:05.204860 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:05.204863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:05.205699 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:05.205710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:05.206309 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:05.206313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:05.206407 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:05.206961 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:05.206972 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:05.209128 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":122,"threads_started":1}
W20260926 11:36:05.209532 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:05.201840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37581 (local address 127.0.98.62:36101)
0926 11:36:05.202393 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:05.202397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:05.202414 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:36:05.202466 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:05.202468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:05.202690 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:05.202904 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:05.203805 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:05.204694 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:05.205834 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:05.206194 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:05.206310 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:05.209284 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37581: BlockingRecv error: recv got EOF from 127.0.98.2:37581 (error 108)
Metrics: {"server-negotiator.queue_time_us":460}
Sep 26 11:36:05 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:05 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:05.416560 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:05.409447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:50085)
0926 11:36:05.409820 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:05.409837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:05.410011 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:05.410306 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:05.410309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:05.410545 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:05.410638 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:05.410643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:05.411467 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:05.411473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:05.412251 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:05.412264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:05.412892 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:05.412895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:05.412992 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:05.413711 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:05.413726 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:05.416352 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":119,"threads_started":1}
W20260926 11:36:05.416615 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:05.409661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50085 (local address 127.0.98.62:36101)
0926 11:36:05.409826 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:05.409833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:05.409957 (+ 124us) server_negotiation.cc:407] Connection header received
0926 11:36:05.410122 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:05.410125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:05.410177 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:05.410280 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:05.410766 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:05.411342 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:05.412475 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:05.412790 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:05.412889 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:05.416517 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50085: BlockingRecv error: recv got EOF from 127.0.98.3:50085 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 11:36:06 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:06 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:06.010200 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:06.003380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:41417)
0926 11:36:06.003762 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:06.003777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:06.003892 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:06.004242 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:06.004245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:06.004468 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:06.004607 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:06.004614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.005433 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.005437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:06.006140 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:06.006152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.006815 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.006819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:06.006899 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:06.007666 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:06.007680 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:06.010038 (+ 2358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":138,"threads_started":1}
W20260926 11:36:06.010612 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:06.003435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41417 (local address 127.0.98.62:36101)
0926 11:36:06.003751 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:06.003754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:06.003893 (+ 139us) server_negotiation.cc:407] Connection header received
0926 11:36:06.003976 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:06.003978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:06.004039 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:06.004153 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:06.004741 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.005297 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.006350 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.006647 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.006767 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:06.010303 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41417: BlockingRecv error: recv got EOF from 127.0.98.1:41417 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":185,"threads_started":1}
Sep 26 11:36:06 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:06 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:06.217346 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:06.210414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:34753)
0926 11:36:06.210790 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:06.210827 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:06.210974 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:06.211267 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:06.211270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:06.211510 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:06.211712 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:06.211724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.212574 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.212580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:06.213194 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:06.213204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.213792 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.213797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:06.213945 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:06.214806 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:06.214827 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:06.217155 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":155,"threads_started":1}
W20260926 11:36:06.217389 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:06.210741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34753 (local address 127.0.98.62:36101)
0926 11:36:06.210964 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:06.210971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:06.210993 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:36:06.211084 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:06.211088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:06.211144 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:06.211247 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:06.211887 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.212431 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.213337 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.213609 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.213729 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:06.217284 (+ 3555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34753: BlockingRecv error: recv got EOF from 127.0.98.2:34753 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 26 11:36:06 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:06 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:06.425156 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:06.417767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:50789)
0926 11:36:06.418347 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:06.418367 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:06.418531 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:06.418911 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:06.418915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:06.419185 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:06.419343 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:06.419350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.420213 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.420217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:06.420985 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:06.420997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.421571 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.421574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:06.421681 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:06.422312 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:06.422326 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:06.425011 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":161,"threads_started":1}
W20260926 11:36:06.425529 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:06.417837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50789 (local address 127.0.98.62:36101)
0926 11:36:06.417992 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:06.417999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:06.418465 (+ 466us) server_negotiation.cc:407] Connection header received
0926 11:36:06.418670 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:06.418673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:06.418725 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:06.418821 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:06.419493 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.420064 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.421170 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:06.421422 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:06.421535 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:06.425264 (+ 3729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50789: BlockingRecv error: recv got EOF from 127.0.98.3:50789 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 11:36:07 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:07 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:07.018754 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:07.011798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:47481)
0926 11:36:07.012180 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:07.012201 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:07.012341 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:07.012624 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:07.012626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:07.012873 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:07.013001 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:07.013008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.013973 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.013977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:07.014653 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:07.014664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.015170 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.015173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:07.015244 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:07.016127 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:07.016145 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:07.018588 (+ 2443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":135,"threads_started":1}
W20260926 11:36:07.019022 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:07.011852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47481 (local address 127.0.98.62:36101)
0926 11:36:07.012248 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:07.012252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:07.012263 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:36:07.012438 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:07.012440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:07.012490 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:07.012580 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:07.013154 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.013815 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.014774 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.015048 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.015149 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:07.018851 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47481: BlockingRecv error: recv got EOF from 127.0.98.1:47481 (error 108)
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":89,"threads_started":1}
Sep 26 11:36:07 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:07 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:07.225344 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:07.218601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:37985)
0926 11:36:07.218961 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:07.218977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:07.219116 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:07.219457 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:07.219459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:07.219761 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:07.219881 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:07.219887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.220782 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.220785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:07.221408 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:07.221417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.221869 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.221872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:07.221938 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:07.222659 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:07.222671 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:07.225194 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":126,"threads_started":1}
W20260926 11:36:07.225589 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:07.218702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37985 (local address 127.0.98.62:36101)
0926 11:36:07.218890 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:07.218896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:07.219119 (+ 223us) server_negotiation.cc:407] Connection header received
0926 11:36:07.219201 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:07.219213 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:07.219314 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:07.219398 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:07.220001 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.220664 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.221524 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.221760 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.221845 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:07.225447 (+ 3602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37985: BlockingRecv error: recv got EOF from 127.0.98.2:37985 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 11:36:07 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:07 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:07.433007 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:07.426310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:33483)
0926 11:36:07.426655 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:07.426692 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:07.426821 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:07.427092 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:07.427094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:07.427326 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:07.427443 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:07.427449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.428468 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.428470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:07.429141 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:07.429151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.429689 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.429692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:07.429790 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:07.430479 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:07.430492 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:07.432794 (+ 2302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":129,"threads_started":1}
W20260926 11:36:07.433050 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:07.426368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33483 (local address 127.0.98.62:36101)
0926 11:36:07.426572 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:07.426579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:07.426786 (+ 207us) server_negotiation.cc:407] Connection header received
0926 11:36:07.426919 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:07.426921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:07.426978 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:07.427062 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:07.427624 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.428337 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.429293 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:07.429541 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:07.429636 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:07.432935 (+ 3299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33483: BlockingRecv error: recv got EOF from 127.0.98.3:33483 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 11:36:08 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:08 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:08.026515 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:08.019848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:36197)
0926 11:36:08.020190 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:08.020205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:08.020363 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:08.020747 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:08.020750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:08.020970 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:08.021083 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:08.021090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.021994 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.021997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:08.022660 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:08.022670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.023237 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.023240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:08.023312 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:08.023975 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:08.023987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:08.026285 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
W20260926 11:36:08.026675 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:08.019925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36197 (local address 127.0.98.62:36101)
0926 11:36:08.020207 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:08.020211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:08.020284 (+ 73us) server_negotiation.cc:407] Connection header received
0926 11:36:08.020547 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:08.020551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:08.020612 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:08.020705 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:08.021226 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.021869 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.022794 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.023081 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.023192 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:08.026513 (+ 3321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36197: BlockingRecv error: recv got EOF from 127.0.98.1:36197 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
Sep 26 11:36:08 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:08 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:08.232445 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:08.226282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:56707)
0926 11:36:08.226625 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:08.226651 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:08.226759 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:08.226990 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:08.226992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:08.227205 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:08.227307 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:08.227313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.228272 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.228275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:08.228885 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:08.228911 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.229416 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.229420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:08.229502 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:08.230192 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:08.230205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:08.232305 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
W20260926 11:36:08.232677 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:08.226390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56707 (local address 127.0.98.62:36101)
0926 11:36:08.226565 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:08.226570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:08.226758 (+ 188us) server_negotiation.cc:407] Connection header received
0926 11:36:08.226822 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:08.226824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:08.226873 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:08.226962 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:08.227474 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.228135 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.229052 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.229272 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.229358 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:08.232524 (+ 3166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56707: BlockingRecv error: recv got EOF from 127.0.98.2:56707 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 11:36:08 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:08 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:08.440156 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:08.434052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:42183)
0926 11:36:08.434455 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:08.434470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:08.434643 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:08.434956 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:08.434959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:08.435186 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:08.435292 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:08.435299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.436218 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.436225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:08.436822 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:08.436830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.437320 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.437324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:08.437432 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:08.437986 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:08.437998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:08.439976 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":71,"threads_started":1}
W20260926 11:36:08.440260 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:08.434126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42183 (local address 127.0.98.62:36101)
0926 11:36:08.434316 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:08.434321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:08.434576 (+ 255us) server_negotiation.cc:407] Connection header received
0926 11:36:08.434761 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:08.434765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:08.434823 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:08.434934 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:08.435433 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.436092 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.436951 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:08.437186 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:08.437300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:08.440144 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42183: BlockingRecv error: recv got EOF from 127.0.98.3:42183 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 11:36:09 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:09 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:09.033784 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:09.027658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:38283)
0926 11:36:09.027948 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:09.027974 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:09.028076 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:09.028413 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:09.028417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:09.028670 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:09.028822 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:09.028831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.029875 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.029878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:09.030521 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:09.030531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.031005 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.031009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:09.031080 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:09.031792 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:09.031811 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:09.033650 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":111,"threads_started":1}
W20260926 11:36:09.034075 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:09.027819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38283 (local address 127.0.98.62:36101)
0926 11:36:09.028098 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:09.028102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:09.028121 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:36:09.028172 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:09.028175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:09.028225 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:09.028343 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:09.028983 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.029763 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.030644 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.030869 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.030967 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:09.033932 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38283: BlockingRecv error: recv got EOF from 127.0.98.1:38283 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":105,"threads_started":1}
Sep 26 11:36:09 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:09 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:09.240065 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:09.233452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:58983)
0926 11:36:09.233745 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:09.233765 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:09.233864 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:09.234205 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:09.234207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:09.234383 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:09.234509 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:09.234514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.235815 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.235818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:09.236647 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:09.236657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.237143 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.237146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:09.237206 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:09.237718 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:09.237730 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:09.239884 (+ 2154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":117,"threads_started":1}
W20260926 11:36:09.240170 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:09.233595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58983 (local address 127.0.98.62:36101)
0926 11:36:09.233781 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:09.233785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:09.233820 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:36:09.234019 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:09.234022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:09.234073 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:09.234167 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:09.234639 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.235659 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.236763 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.237029 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.237117 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:09.240054 (+ 2937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58983: BlockingRecv error: recv got EOF from 127.0.98.2:58983 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 11:36:09 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:09 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:09.447830 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:09.441314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:51963)
0926 11:36:09.441690 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:09.441707 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:09.441832 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:09.442192 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:09.442195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:09.442472 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:09.442595 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:09.442602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.443499 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.443502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:09.444173 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:09.444187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.444747 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.444750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:09.444823 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:09.445533 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:09.445545 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:09.447705 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
W20260926 11:36:09.448211 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:09.441469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51963 (local address 127.0.98.62:36101)
0926 11:36:09.441622 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:09.441626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:09.441817 (+ 191us) server_negotiation.cc:407] Connection header received
0926 11:36:09.441978 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:09.441981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:09.442038 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:09.442137 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:09.442732 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.443345 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.444333 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:09.444606 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:09.444709 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:09.447895 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51963: BlockingRecv error: recv got EOF from 127.0.98.3:51963 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 11:36:10 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:10 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:10.045446 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:10.034672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:34121)
0926 11:36:10.035024 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:10.035048 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:10.035184 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:10.035612 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:10.035616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:10.035810 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:10.035907 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:10.035913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.036701 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.036704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:10.037250 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:10.037258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.037880 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.037882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:10.037942 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:10.042866 (+ 4924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:10.042923 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:10.045306 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":92,"threads_started":1}
W20260926 11:36:10.045753 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:10.034714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34121 (local address 127.0.98.62:36101)
0926 11:36:10.035002 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:10.035007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:10.035121 (+ 114us) server_negotiation.cc:407] Connection header received
0926 11:36:10.035359 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:10.035366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:10.035427 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:10.035548 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:10.036040 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.036595 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.037467 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.037775 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.037860 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:10.045602 (+ 7742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34121: BlockingRecv error: recv got EOF from 127.0.98.1:34121 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":63,"threads_started":1}
Sep 26 11:36:10 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:10 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:10.248262 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:10.241176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:36989)
0926 11:36:10.241509 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:10.241528 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:10.241662 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:10.242024 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:10.242045 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:10.242278 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:10.242387 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:10.242393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.243808 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.243812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:10.244520 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:10.244531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.245118 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.245121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:10.245194 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:10.245923 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:10.245936 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:10.248111 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
W20260926 11:36:10.248608 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:10.241248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36989 (local address 127.0.98.62:36101)
0926 11:36:10.241392 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:10.241397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:10.241661 (+ 264us) server_negotiation.cc:407] Connection header received
0926 11:36:10.241754 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:10.241757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:10.241820 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:10.241947 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:10.242633 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.243231 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.244686 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.244968 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.245061 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:10.248446 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36989: BlockingRecv error: recv got EOF from 127.0.98.2:36989 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 11:36:10 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:10 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:10.455667 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:10.448773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:33807)
0926 11:36:10.449185 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:10.449198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:10.449338 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:10.449661 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:10.449664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:10.449882 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:10.449977 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:10.449983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.450785 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.450790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:10.451654 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:10.451671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.452237 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.452241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:10.452335 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:10.452937 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:10.452951 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:10.455459 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":87,"threads_started":1}
W20260926 11:36:10.455902 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:10.448836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33807 (local address 127.0.98.62:36101)
0926 11:36:10.448978 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:10.448983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:10.449311 (+ 328us) server_negotiation.cc:407] Connection header received
0926 11:36:10.449482 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:10.449488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:10.449549 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:10.449635 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:10.450098 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.450670 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.451816 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:10.452101 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:10.452229 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:10.455771 (+ 3542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33807: BlockingRecv error: recv got EOF from 127.0.98.3:33807 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 11:36:11 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:11 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:11.054473 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:11.046742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:56849)
0926 11:36:11.047121 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:11.047137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:11.047272 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:11.047717 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:11.047723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:11.048056 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:11.048205 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:11.048213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.049336 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.049340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:11.050259 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:11.050273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.050919 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.050922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:11.051028 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:11.051834 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:11.051854 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:11.054299 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
W20260926 11:36:11.054723 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:11.046952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56849 (local address 127.0.98.62:36101)
0926 11:36:11.047217 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:11.047221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:11.047238 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:36:11.047385 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:11.047387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:11.047438 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:11.047531 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:11.048383 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.049118 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.050386 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.050728 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.050886 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:11.054536 (+ 3650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56849: BlockingRecv error: recv got EOF from 127.0.98.1:56849 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
Sep 26 11:36:11 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:11 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:11.256547 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:11.249604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:36251)
0926 11:36:11.250064 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:11.250105 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:11.250244 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:11.250608 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:11.250611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:11.250885 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:11.251026 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:11.251037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.252184 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.252187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:11.252816 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:11.252826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.253392 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.253395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:11.253474 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:11.254253 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:11.254267 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:11.256387 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":153,"threads_started":1}
W20260926 11:36:11.256860 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:11.249704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36251 (local address 127.0.98.62:36101)
0926 11:36:11.249943 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:11.249948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:11.250204 (+ 256us) server_negotiation.cc:407] Connection header received
0926 11:36:11.250374 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:11.250377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:11.250438 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:11.250535 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:11.251246 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.252043 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.252961 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.253234 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.253343 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:11.256708 (+ 3365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36251: BlockingRecv error: recv got EOF from 127.0.98.2:36251 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 11:36:11 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:11 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:11.464309 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:11.456832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:54159)
0926 11:36:11.457209 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:11.457225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:11.457335 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:11.457657 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:11.457660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:11.457912 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:11.458034 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:11.458041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.458940 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.458943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:11.459772 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:11.459783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.460317 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.460320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:11.460392 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:11.461413 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:11.461434 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:11.464065 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":135,"threads_started":1}
W20260926 11:36:11.464532 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:11.457017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54159 (local address 127.0.98.62:36101)
0926 11:36:11.457257 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:11.457263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:11.457285 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:36:11.457459 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:11.457461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:11.457517 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:11.457599 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:11.458161 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.458807 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.459897 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:11.460151 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:11.460281 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:11.464346 (+ 4065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54159: BlockingRecv error: recv got EOF from 127.0.98.3:54159 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 11:36:12 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:12 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:12.063642 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:12.055816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:48397)
0926 11:36:12.056201 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:12.056218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:12.056348 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:12.056823 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:12.056826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:12.057070 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:12.057264 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:12.057275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.058349 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.058360 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:12.059051 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:12.059061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.059921 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.059926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:12.060050 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:12.060815 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:12.060830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:12.063390 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
W20260926 11:36:12.063920 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:12.055863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48397 (local address 127.0.98.62:36101)
0926 11:36:12.056216 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:12.056222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:12.056307 (+ 85us) server_negotiation.cc:407] Connection header received
0926 11:36:12.056452 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:12.056456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:12.056518 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:12.056646 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:12.057397 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.058187 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.059336 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.059742 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.059851 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:12.063700 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48397: BlockingRecv error: recv got EOF from 127.0.98.1:48397 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
Sep 26 11:36:12 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:12 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:12.265038 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:12.257668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:50219)
0926 11:36:12.258119 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:12.258139 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:12.258334 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:12.258732 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:12.258734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:12.259066 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:12.259215 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:12.259222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.260326 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.260329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:12.260969 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:12.260977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.261580 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.261584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:12.261672 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:12.262394 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:12.262407 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:12.264821 (+ 2414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":110,"threads_started":1}
W20260926 11:36:12.265136 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:12.257906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50219 (local address 127.0.98.62:36101)
0926 11:36:12.258133 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:12.258137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:12.258280 (+ 143us) server_negotiation.cc:407] Connection header received
0926 11:36:12.258479 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:12.258483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:12.258559 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:12.258686 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:12.259389 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.260133 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.261109 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.261413 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.261574 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:12.265001 (+ 3427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50219: BlockingRecv error: recv got EOF from 127.0.98.2:50219 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 11:36:12 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:12 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:12.473835 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:12.465598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:56651)
0926 11:36:12.465995 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:12.466014 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:12.466177 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:12.466702 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:12.466704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:12.466952 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:12.467093 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:12.467100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.468232 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.468236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:12.469082 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:12.469095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.469916 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.469924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:12.470042 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:12.470961 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:12.470982 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:12.473649 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":107,"threads_started":1}
W20260926 11:36:12.474004 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:12.465792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56651 (local address 127.0.98.62:36101)
0926 11:36:12.466016 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:12.466021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:12.466187 (+ 166us) server_negotiation.cc:407] Connection header received
0926 11:36:12.466436 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:12.466440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:12.466506 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:12.466612 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:12.467321 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.468025 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.469277 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:12.469734 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:12.469903 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:12.473841 (+ 3938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56651: BlockingRecv error: recv got EOF from 127.0.98.3:56651 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 11:36:13 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:13 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:13.072234 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:13.064858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:32999)
0926 11:36:13.065275 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:13.065293 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:13.065427 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:13.065776 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:13.065779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:13.065980 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:13.066096 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:13.066101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.067241 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.067248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:13.068047 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:13.068057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.068774 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.068777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:13.068844 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:13.069533 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:13.069544 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:13.072014 (+ 2470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
W20260926 11:36:13.072376 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:13.065025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32999 (local address 127.0.98.62:36101)
0926 11:36:13.065350 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:13.065355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:13.065376 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:36:13.065552 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:13.065555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:13.065614 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:13.065731 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:13.066296 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.067069 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.068257 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.068643 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.068758 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:13.072230 (+ 3472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32999: BlockingRecv error: recv got EOF from 127.0.98.1:32999 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":75,"threads_started":1}
Sep 26 11:36:13 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:13 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:13.272959 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:13.266142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:48281)
0926 11:36:13.266530 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:13.266544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:13.266703 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:13.267023 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:13.267026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:13.267301 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:13.267426 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:13.267432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.268314 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.268319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:13.268950 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:13.268959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.269456 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.269460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:13.269550 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:13.270201 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:13.270213 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:13.272749 (+ 2536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
W20260926 11:36:13.273084 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:13.266211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48281 (local address 127.0.98.62:36101)
0926 11:36:13.266434 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:13.266439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:13.266633 (+ 194us) server_negotiation.cc:407] Connection header received
0926 11:36:13.266832 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:13.266835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:13.266896 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:13.266997 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:13.267556 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.268189 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.269078 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.269321 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.269438 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:13.272955 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48281: BlockingRecv error: recv got EOF from 127.0.98.2:48281 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 26 11:36:13 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:13 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:13.483048 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:13.475214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:47269)
0926 11:36:13.475896 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:13.475914 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:13.476132 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:13.476582 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:13.476586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:13.476896 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:13.477013 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:13.477020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.478058 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.478061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:13.478785 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:13.478795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.479336 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.479339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:13.479408 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:13.480154 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:13.480175 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:13.482876 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":166,"threads_started":1}
W20260926 11:36:13.483355 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:13.475463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47269 (local address 127.0.98.62:36101)
0926 11:36:13.475684 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:13.475693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:13.476019 (+ 326us) server_negotiation.cc:407] Connection header received
0926 11:36:13.476333 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:13.476340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:13.476417 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:13.476543 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:13.477167 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.477919 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.478909 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:13.479160 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:13.479261 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:13.483145 (+ 3884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47269: BlockingRecv error: recv got EOF from 127.0.98.3:47269 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 11:36:14 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:14 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:14.081244 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:14.073571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:37357)
0926 11:36:14.073907 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:14.073923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:14.074061 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:14.074360 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:14.074362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:14.074585 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:14.074708 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:14.074742 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.075814 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.075818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:14.076781 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:14.076793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.077369 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.077372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:14.077463 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:14.078421 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:14.078437 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:14.081072 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
W20260926 11:36:14.081473 9626 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:14.073610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37357 (local address 127.0.98.62:36101)
0926 11:36:14.073927 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:14.073931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:14.074013 (+ 82us) server_negotiation.cc:407] Connection header received
0926 11:36:14.074157 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:14.074159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:14.074210 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:14.074324 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:14.074887 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.075660 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.076941 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.077233 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.077364 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:14.081290 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37357: BlockingRecv error: recv got EOF from 127.0.98.1:37357 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":144,"threads_started":1}
Sep 26 11:36:14 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:14 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:14.280920 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:14.274349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:51359)
0926 11:36:14.274844 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:14.274862 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:14.275003 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:14.275283 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:14.275286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:14.275565 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:14.275764 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:14.275774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.276599 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.276604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:14.277210 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:14.277218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.277706 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.277709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:14.277791 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:14.278482 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:14.278503 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:14.280711 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
W20260926 11:36:14.281031 9626 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:14.274572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51359 (local address 127.0.98.62:36101)
0926 11:36:14.274744 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:14.274748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:14.274948 (+ 200us) server_negotiation.cc:407] Connection header received
0926 11:36:14.275110 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:14.275114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:14.275171 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:14.275263 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:14.275871 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.276479 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.277334 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.277583 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.277705 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:14.280896 (+ 3191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51359: BlockingRecv error: recv got EOF from 127.0.98.2:51359 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 11:36:14 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:14 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:14.491304 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:14.484258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:35333)
0926 11:36:14.484572 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:14.484586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:14.484712 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:14.485036 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:14.485038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:14.485367 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:14.485516 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:14.485526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.486681 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.486684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:14.487641 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:14.487653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.488167 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.488170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:14.488246 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:14.488871 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:14.488884 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:14.491094 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
W20260926 11:36:14.491493 9626 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:14.484345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35333 (local address 127.0.98.62:36101)
0926 11:36:14.484544 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:14.484548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:14.484671 (+ 123us) server_negotiation.cc:407] Connection header received
0926 11:36:14.484850 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:14.484853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:14.484913 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:14.485005 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:14.485685 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.486524 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.487775 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:14.488040 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:14.488136 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:14.491305 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35333: BlockingRecv error: recv got EOF from 127.0.98.3:35333 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 26 11:36:15 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:15 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:15.092718 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:15.082361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:33917)
0926 11:36:15.082778 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:15.082795 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:15.082915 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:15.083219 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:15.083222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:15.083455 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:15.083600 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:15.083606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.084597 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.084600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:15.085498 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:15.085509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.086166 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.086171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:15.086268 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:15.090174 (+ 3906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:15.090194 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:15.092535 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
W20260926 11:36:15.092978 9630 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:15.082447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33917 (local address 127.0.98.62:36101)
0926 11:36:15.082879 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:15.082883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:15.082900 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:36:15.083017 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:15.083020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:15.083070 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:15.083164 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:15.083712 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.084453 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.085634 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.086021 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.086121 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:15.092788 (+ 6667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33917: BlockingRecv error: recv got EOF from 127.0.98.1:33917 (error 108)
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":116,"threads_started":1}
Sep 26 11:36:15 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:15 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:15.288468 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:15.281886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:54327)
0926 11:36:15.282243 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:15.282257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:15.282359 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:15.282660 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:15.282662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:15.282866 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:15.282987 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:15.282993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.284057 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.284060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:15.284732 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:15.284742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.285260 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.285264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:15.285338 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:15.285939 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:15.285954 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:15.288305 (+ 2351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":133,"threads_started":1}
W20260926 11:36:15.288662 9630 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:15.281950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54327 (local address 127.0.98.62:36101)
0926 11:36:15.282123 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:15.282128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:15.282334 (+ 206us) server_negotiation.cc:407] Connection header received
0926 11:36:15.282493 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:15.282496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:15.282545 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:15.282619 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:15.283120 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.283914 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.284866 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.285125 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.285217 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:15.288500 (+ 3283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54327: BlockingRecv error: recv got EOF from 127.0.98.2:54327 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 11:36:15 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:15 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:15.499009 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:15.492280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:57109)
0926 11:36:15.492673 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:15.492686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:15.492790 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:15.493078 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:15.493080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:15.493307 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:15.493468 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:15.493476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.494347 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.494350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:15.495068 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:15.495078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.495745 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.495748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:15.495816 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:15.496443 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:15.496455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:15.498810 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":201,"threads_started":1}
W20260926 11:36:15.499155 9630 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:15.492480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57109 (local address 127.0.98.62:36101)
0926 11:36:15.492641 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:15.492645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:15.492793 (+ 148us) server_negotiation.cc:407] Connection header received
0926 11:36:15.492874 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:15.492876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:15.492930 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:15.493002 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:15.493613 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.494217 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.495231 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:15.495510 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:15.495726 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:15.499005 (+ 3279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57109: BlockingRecv error: recv got EOF from 127.0.98.3:57109 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 11:36:16 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:16 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:16.100616 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:16.093952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:48003)
0926 11:36:16.094260 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:16.094275 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:16.094438 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:16.094915 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:16.094919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:16.095182 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:16.095305 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:16.095311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.096412 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.096416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:16.097074 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:16.097083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.097667 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.097670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:16.097907 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:16.098580 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:16.098592 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:16.100488 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":115,"threads_started":1}
W20260926 11:36:16.100879 9634 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:16.094045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48003 (local address 127.0.98.62:36101)
0926 11:36:16.094265 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:16.094270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:16.094370 (+ 100us) server_negotiation.cc:407] Connection header received
0926 11:36:16.094693 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:16.094695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:16.094736 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:16.094817 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:16.095520 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.096171 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.097217 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.097546 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.097644 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:16.100721 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48003: BlockingRecv error: recv got EOF from 127.0.98.1:48003 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
Sep 26 11:36:16 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:36:16 dist-test-slave-4pw5 krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:36:16.297289 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:16.289911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:41649)
0926 11:36:16.290289 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:16.290310 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:16.290462 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:16.290809 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:16.290812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:16.291097 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:16.291251 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:16.291260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.292404 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.292408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:16.293249 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:16.293259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.293748 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.293751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:16.293818 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:16.294441 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:16.294453 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:16.297068 (+ 2615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36101: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
W20260926 11:36:16.297523 9634 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:16.290029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41649 (local address 127.0.98.62:36101)
0926 11:36:16.290297 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:16.290305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:16.290390 (+ 85us) server_negotiation.cc:407] Connection header received
0926 11:36:16.290612 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:16.290616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:16.290684 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:16.290790 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:16.291441 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.292248 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.293390 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.293622 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.293741 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:16.297331 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41649: BlockingRecv error: recv got EOF from 127.0.98.2:41649 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
I20260926 11:36:16.365078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 9159
I20260926 11:36:16.372094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 9311
I20260926 11:36:16.378328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 9443
I20260926 11:36:16.384650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 9089
2026-09-26T11:36:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18295 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T11:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:36:16Z Disabled control of system clock
I20260926 11:36:16.421514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38279
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 11:36:16.558797 9639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:16.559224 9639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:16.559268 9639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:16.564123 9639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:36:16.564249 9639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:16.564266 9639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:16.564286 9639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:36:16.564304 9639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:36:16.570003 9639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:16.571565 9639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:16.573520 9639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:16.584271 9645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:16.584177 9647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:16.584175 9644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:16.584254 9639 server_base.cc:1034] running on GCE node
I20260926 11:36:16.585332 9639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:16.586889 9639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:16.588094 9639 hybrid_clock.cc:648] HybridClock initialized: now 1790422576588067 us; error 54 us; skew 500 ppm
I20260926 11:36:16.590734 9639 webserver.cc:460] Webserver started at http://127.0.98.62:46101/ using document root <none> and password file <none>
I20260926 11:36:16.591706 9639 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:16.591790 9639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:16.592059 9639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:16.594759 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "0a9f554fe0ec497fb7a65b0f8755ce55"
format_stamp: "Formatted at 2026-09-26 11:36:16 on dist-test-slave-4pw5"
server_key: "59389e2b826bc191e8800c766ea0b151"
server_key_iv: "735c6703c1eb1bbcb0211970639c556f"
server_key_version: "encryptionkey@0"
I20260926 11:36:16.595508 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a9f554fe0ec497fb7a65b0f8755ce55"
format_stamp: "Formatted at 2026-09-26 11:36:16 on dist-test-slave-4pw5"
server_key: "59389e2b826bc191e8800c766ea0b151"
server_key_iv: "735c6703c1eb1bbcb0211970639c556f"
server_key_version: "encryptionkey@0"
I20260926 11:36:16.600593 9639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 11:36:16.604091 9653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:16.605871 9639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:36:16.606077 9639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "0a9f554fe0ec497fb7a65b0f8755ce55"
format_stamp: "Formatted at 2026-09-26 11:36:16 on dist-test-slave-4pw5"
server_key: "59389e2b826bc191e8800c766ea0b151"
server_key_iv: "735c6703c1eb1bbcb0211970639c556f"
server_key_version: "encryptionkey@0"
I20260926 11:36:16.606263 9639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:16.636793 9639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:16.637840 9639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:16.638108 9639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:16.647096 9639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38279
I20260926 11:36:16.647102 9705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38279 every 8 connection(s)
I20260926 11:36:16.648483 9639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:36:16.650429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 9639
I20260926 11:36:16.650569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:36:16.650951 392 external_mini_cluster.cc:1428] Setting key 7312b401a841ebbbc2aa265c448a9b7b
I20260926 11:36:16.653787 9706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:36:16.661929 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55: Bootstrap starting.
I20260926 11:36:16.662909 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:16.652971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.0.98.62:38279)
0926 11:36:16.653878 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:16.653913 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:16.653973 (+ 60us) server_negotiation.cc:407] Connection header received
0926 11:36:16.654789 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:16.654814 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:16.655368 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:16.655839 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:16.656409 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.657522 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.658504 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.659075 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.659274 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:16.660485 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:16.660490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:16.660509 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:36:16.661058 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:16.661137 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:36:16.661162 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:16.661287 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:16.661457 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 11:36:16.661841 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":206,"threads_started":1}
I20260926 11:36:16.665414 9706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55: Neither blocks nor log segments found. Creating new log.
I20260926 11:36:16.666488 9706 log.cc:824] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55: Log is configured to *not* fsync() on all Append() calls
I20260926 11:36:16.669009 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55: No bootstrap required, opened a new log
I20260926 11:36:16.673319 9706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9f554fe0ec497fb7a65b0f8755ce55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38279 } }
I20260926 11:36:16.673636 9706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:36:16.673697 9706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9f554fe0ec497fb7a65b0f8755ce55, State: Initialized, Role: FOLLOWER
I20260926 11:36:16.674350 9706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9f554fe0ec497fb7a65b0f8755ce55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38279 } }
I20260926 11:36:16.674539 9706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:36:16.674647 9706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:36:16.674787 9706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:36:16.676064 9706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9f554fe0ec497fb7a65b0f8755ce55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38279 } }
I20260926 11:36:16.676497 9706 leader_election.cc:302] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a9f554fe0ec497fb7a65b0f8755ce55; no voters:
I20260926 11:36:16.676903 9706 leader_election.cc:288] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:36:16.677100 9711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:36:16.677493 9711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [term 1 LEADER]: Becoming Leader. State: Replica: 0a9f554fe0ec497fb7a65b0f8755ce55, State: Running, Role: LEADER
I20260926 11:36:16.677942 9711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9f554fe0ec497fb7a65b0f8755ce55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38279 } }
I20260926 11:36:16.678478 9706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:36:16.680080 9713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a9f554fe0ec497fb7a65b0f8755ce55. Latest consensus state: current_term: 1 leader_uuid: "0a9f554fe0ec497fb7a65b0f8755ce55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9f554fe0ec497fb7a65b0f8755ce55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38279 } } }
I20260926 11:36:16.680119 9712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a9f554fe0ec497fb7a65b0f8755ce55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9f554fe0ec497fb7a65b0f8755ce55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38279 } } }
I20260926 11:36:16.680250 9713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [sys.catalog]: This master's current role is: LEADER
I20260926 11:36:16.680249 9712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55 [sys.catalog]: This master's current role is: LEADER
I20260926 11:36:16.680723 9721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:36:16.684687 9721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:36:16.692229 9721 catalog_manager.cc:1347] Generated new cluster ID: 903c798b33134793856c568ee79994ab
I20260926 11:36:16.692334 9721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:36:16.704571 9721 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:36:16.705904 9721 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:36:16.715533 9721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a9f554fe0ec497fb7a65b0f8755ce55: Generated new TSK 0
I20260926 11:36:16.716606 9721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:36:16.722735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38279
--builtin_ntp_servers=127.0.98.20:35343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 11:36:16.859740 9730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:16.860181 9730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:16.860265 9730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:16.865020 9730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:16.865127 9730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:16.865198 9730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:36:16.870960 9730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:16.872534 9730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:16.874297 9730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:16.886765 9735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:16.886744 9738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:16.887136 9730 server_base.cc:1034] running on GCE node
W20260926 11:36:16.887333 9736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:16.887893 9730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:16.888692 9730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:16.889894 9730 hybrid_clock.cc:648] HybridClock initialized: now 1790422576889869 us; error 34 us; skew 500 ppm
I20260926 11:36:16.892601 9730 webserver.cc:460] Webserver started at http://127.0.98.1:38643/ using document root <none> and password file <none>
I20260926 11:36:16.893317 9730 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:16.893388 9730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:16.893596 9730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:16.896291 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "f0c47c9aec2444e980d205f2e6b0fd7e"
format_stamp: "Formatted at 2026-09-26 11:36:16 on dist-test-slave-4pw5"
server_key: "580a0eb3c200a8619ded61467660344e"
server_key_iv: "a4be536b91bc0a0919d8864b77912ed3"
server_key_version: "encryptionkey@0"
I20260926 11:36:16.896939 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f0c47c9aec2444e980d205f2e6b0fd7e"
format_stamp: "Formatted at 2026-09-26 11:36:16 on dist-test-slave-4pw5"
server_key: "580a0eb3c200a8619ded61467660344e"
server_key_iv: "a4be536b91bc0a0919d8864b77912ed3"
server_key_version: "encryptionkey@0"
I20260926 11:36:16.901856 9730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:36:16.905110 9744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:16.906716 9730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:36:16.906875 9730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "f0c47c9aec2444e980d205f2e6b0fd7e"
format_stamp: "Formatted at 2026-09-26 11:36:16 on dist-test-slave-4pw5"
server_key: "580a0eb3c200a8619ded61467660344e"
server_key_iv: "a4be536b91bc0a0919d8864b77912ed3"
server_key_version: "encryptionkey@0"
I20260926 11:36:16.907011 9730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:16.920008 9730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:16.920923 9730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:16.921144 9730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:16.921844 9730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:16.923000 9730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:16.923071 9730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:16.923147 9730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:16.923194 9730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:16.933349 9730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35177
I20260926 11:36:16.933398 9857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35177 every 8 connection(s)
I20260926 11:36:16.934535 9730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:36:16.942082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 9730
I20260926 11:36:16.942310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:36:16.942706 392 external_mini_cluster.cc:1428] Setting key 72202499e82a824bb7c74b6c5c4a1e64
I20260926 11:36:16.945358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38279
--builtin_ntp_servers=127.0.98.20:35343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 11:36:16.947450 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:16.937147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58323 (local address 127.0.98.62:38279)
0926 11:36:16.937344 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:16.937350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:16.938843 (+ 1493us) server_negotiation.cc:407] Connection header received
0926 11:36:16.940747 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:16.940751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:16.940843 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:16.940966 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:16.942211 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.942974 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.944909 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.945247 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.945373 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:16.946809 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:16.946813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:16.946817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:36:16.946837 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:16.946861 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:36:16.946870 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:16.946951 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:16.947146 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 11:36:16.947269 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 11:36:16.948629 9860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:16.937714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38279 (local address 127.0.98.1:58323)
0926 11:36:16.938608 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:16.938666 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:16.940459 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:16.941213 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:16.941224 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:16.941343 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:16.942039 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:16.942059 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.943117 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.943121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:16.944765 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:16.944776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:16.945434 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:16.945439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:16.945687 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:16.946479 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:36:16.946481 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:16.946535 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:36:16.946622 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:16.946966 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:16.946978 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:16.946985 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:36:16.947137 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 11:36:16.947522 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":138,"threads_started":1}
I20260926 11:36:16.950359 9858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38279
I20260926 11:36:16.950682 9858 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:16.951434 9858 heartbeater.cc:511] Master 127.0.98.62:38279 requested a full tablet report, sending...
I20260926 11:36:16.953972 9668 ts_manager.cc:194] Registered new tserver with Master: f0c47c9aec2444e980d205f2e6b0fd7e (127.0.98.1:35177)
I20260926 11:36:16.956363 9668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58323
W20260926 11:36:17.083726 9861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:17.084072 9861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:17.084129 9861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:17.088574 9861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:17.088668 9861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:17.088734 9861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:36:17.094182 9861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:17.095680 9861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:17.097414 9861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:17.109019 9869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:17.109019 9866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:17.109284 9861 server_base.cc:1034] running on GCE node
W20260926 11:36:17.109090 9867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:17.111172 9861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:17.112146 9861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:17.113327 9861 hybrid_clock.cc:648] HybridClock initialized: now 1790422577113303 us; error 46 us; skew 500 ppm
I20260926 11:36:17.115995 9861 webserver.cc:460] Webserver started at http://127.0.98.2:34661/ using document root <none> and password file <none>
I20260926 11:36:17.116823 9861 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:17.116901 9861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:17.117165 9861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:17.119910 9861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "d9dbcd7c323a4db3b88852b9e0bbbd1c"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "62d96055ee48a29f4939e16f7493e4b9"
server_key_iv: "2fd0cd48cd4bdd04703de5c6986f06ab"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.120609 9861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d9dbcd7c323a4db3b88852b9e0bbbd1c"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "62d96055ee48a29f4939e16f7493e4b9"
server_key_iv: "2fd0cd48cd4bdd04703de5c6986f06ab"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.126097 9861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:36:17.129760 9875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.131708 9861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:17.131875 9861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "d9dbcd7c323a4db3b88852b9e0bbbd1c"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "62d96055ee48a29f4939e16f7493e4b9"
server_key_iv: "2fd0cd48cd4bdd04703de5c6986f06ab"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.132048 9861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:17.155807 9861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:17.156865 9861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:17.157125 9861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:17.157977 9861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:17.159302 9861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:17.159389 9861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.159474 9861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:17.159519 9861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.171440 9861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37829
I20260926 11:36:17.171473 9988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37829 every 8 connection(s)
I20260926 11:36:17.172755 9861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:36:17.174746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 9861
I20260926 11:36:17.174892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:36:17.175257 392 external_mini_cluster.cc:1428] Setting key 48f34a7fc46288b56313cb455eb9ce93
I20260926 11:36:17.177788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38279
--builtin_ntp_servers=127.0.98.20:35343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 11:36:17.186853 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:17.175165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60501 (local address 127.0.98.62:38279)
0926 11:36:17.175353 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:17.175359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:17.176961 (+ 1602us) server_negotiation.cc:407] Connection header received
0926 11:36:17.179811 (+ 2850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:17.179815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:17.179943 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:17.180037 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:17.181684 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:17.182613 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:17.184218 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:17.184641 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:17.184772 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:17.186276 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:17.186280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:17.186282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:36:17.186298 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:17.186321 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:36:17.186330 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:17.186402 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:17.186642 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 11:36:17.186711 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 11:36:17.188516 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:17.175863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38279 (local address 127.0.98.2:60501)
0926 11:36:17.176766 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:17.176868 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:17.179345 (+ 2477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:17.180361 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:17.180376 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:17.180555 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:17.181491 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:17.181511 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:17.182777 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:17.182780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:17.184046 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:17.184059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:17.184840 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:17.184844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:17.185030 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:17.185854 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:36:17.185858 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:17.185934 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:36:17.186066 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:17.186425 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:17.186438 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:17.186447 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 11:36:17.186617 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 11:36:17.187046 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":166,"threads_started":1}
I20260926 11:36:17.189931 9989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38279
I20260926 11:36:17.190261 9989 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:17.191087 9989 heartbeater.cc:511] Master 127.0.98.62:38279 requested a full tablet report, sending...
I20260926 11:36:17.192634 9668 ts_manager.cc:194] Registered new tserver with Master: d9dbcd7c323a4db3b88852b9e0bbbd1c (127.0.98.2:37829)
I20260926 11:36:17.193485 9668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60501
W20260926 11:36:17.314842 9992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:17.315142 9992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:17.315181 9992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:17.319273 9992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:17.319345 9992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:17.319406 9992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:36:17.324400 9992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:17.325568 9992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:17.327077 9992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:17.337531 9997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:17.337692 9992 server_base.cc:1034] running on GCE node
W20260926 11:36:17.337550 9998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:17.337839 10000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:17.338366 9992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:17.339134 9992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:17.340343 9992 hybrid_clock.cc:648] HybridClock initialized: now 1790422577340323 us; error 36 us; skew 500 ppm
I20260926 11:36:17.342545 9992 webserver.cc:460] Webserver started at http://127.0.98.3:44217/ using document root <none> and password file <none>
I20260926 11:36:17.343235 9992 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:17.343334 9992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:17.343662 9992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:17.346148 9992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "fe90bb4246084470873151d1801ea346"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "3ab18150608fe62908f6c9daa78ed3cd"
server_key_iv: "9f20b8152b479dc65a14cebae2589174"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.346782 9992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fe90bb4246084470873151d1801ea346"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "3ab18150608fe62908f6c9daa78ed3cd"
server_key_iv: "9f20b8152b479dc65a14cebae2589174"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.351140 9992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 11:36:17.354182 10006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.355835 9992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:17.355993 9992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "fe90bb4246084470873151d1801ea346"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "3ab18150608fe62908f6c9daa78ed3cd"
server_key_iv: "9f20b8152b479dc65a14cebae2589174"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.356129 9992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:17.368423 9992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:17.369213 9992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:17.369449 9992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:17.370098 9992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:17.371152 9992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:17.371230 9992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.371306 9992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:17.371352 9992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.381788 9992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37469
I20260926 11:36:17.382091 10119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37469 every 8 connection(s)
I20260926 11:36:17.383234 9992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:36:17.386925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 9992
I20260926 11:36:17.387077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:36:17.387499 392 external_mini_cluster.cc:1428] Setting key 109bab7a4aa5cc0322dce3f08da4f9e7
I20260926 11:36:17.397079 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:17.391715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56783 (local address 127.0.98.62:38279)
0926 11:36:17.391880 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:17.391885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:17.391905 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:36:17.391953 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:17.391956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:17.392015 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:17.392120 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:17.393195 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:17.393744 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:17.394995 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:17.395254 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:17.395348 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:17.396592 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:17.396595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:17.396597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:36:17.396612 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:17.396631 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:36:17.396640 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:17.396723 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:17.396888 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 11:36:17.396951 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:36:17.398057 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:17.385977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38279 (local address 127.0.98.3:56783)
0926 11:36:17.386781 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:17.386838 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:17.388618 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:17.392341 (+ 3723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:17.392353 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:17.392475 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:17.393043 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:17.393057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:17.393896 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:17.393899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:17.394832 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:17.394843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:17.395403 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:17.395406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:17.395663 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:17.396313 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:36:17.396314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:17.396357 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:36:17.396429 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:17.396738 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:17.396753 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:17.396757 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 11:36:17.396871 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 11:36:17.397175 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
I20260926 11:36:17.399526 10120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38279
I20260926 11:36:17.399899 10120 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:17.400653 10120 heartbeater.cc:511] Master 127.0.98.62:38279 requested a full tablet report, sending...
I20260926 11:36:17.401919 9668 ts_manager.cc:194] Registered new tserver with Master: fe90bb4246084470873151d1801ea346 (127.0.98.3:37469)
I20260926 11:36:17.402555 9668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56783
I20260926 11:36:17.403314 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 11:36:17.407294 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:17.406139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.0.98.62:38279)
0926 11:36:17.406280 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:17.406285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:17.406562 (+ 277us) server_negotiation.cc:407] Connection header received
0926 11:36:17.406669 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:17.406672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:17.406713 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:17.406807 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:17.407141 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59822: BlockingRecv error: recv got EOF from 127.0.0.1:59822 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:36:17.407501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 9730
I20260926 11:36:17.414086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 9861
I20260926 11:36:17.420316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 9992
I20260926 11:36:17.426024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 9639
2026-09-26T11:36:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1037 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T11:36:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:36:17Z Disabled control of system clock
I20260926 11:36:17.457722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40079
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 11:36:17.574635 10131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:17.575002 10131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:17.575088 10131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:17.579309 10131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:36:17.579427 10131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:17.579456 10131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:17.579499 10131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:36:17.579533 10131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:36:17.584630 10131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:17.585821 10131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:17.587270 10131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:17.596469 10136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:17.596472 10137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:17.596469 10139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:17.597107 10131 server_base.cc:1034] running on GCE node
I20260926 11:36:17.597734 10131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:17.599040 10131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:17.600242 10131 hybrid_clock.cc:648] HybridClock initialized: now 1790422577600200 us; error 65 us; skew 500 ppm
I20260926 11:36:17.602501 10131 webserver.cc:460] Webserver started at http://127.0.98.62:40837/ using document root <none> and password file <none>
I20260926 11:36:17.603159 10131 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:17.603335 10131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:17.603688 10131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:17.606071 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "2d0a4455befac0a5da849e31476d7813"
server_key_iv: "32d888c9c3d36a5a9e3cbb03ffdce11e"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.606721 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "2d0a4455befac0a5da849e31476d7813"
server_key_iv: "32d888c9c3d36a5a9e3cbb03ffdce11e"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.611740 10131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:36:17.614612 10145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.616261 10131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:17.616483 10131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "2d0a4455befac0a5da849e31476d7813"
server_key_iv: "32d888c9c3d36a5a9e3cbb03ffdce11e"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.616637 10131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:17.621922 10131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:17.622658 10131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:17.622888 10131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:17.630921 10131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40079
I20260926 11:36:17.630944 10197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40079 every 8 connection(s)
I20260926 11:36:17.632082 10131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:36:17.634222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 10131
I20260926 11:36:17.634356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:36:17.634804 392 external_mini_cluster.cc:1428] Setting key 07206e7f94d0ea8ff0aeb41b6d475239
I20260926 11:36:17.636337 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:36:17.640957 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:17.636632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.0.98.62:40079)
0926 11:36:17.637348 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:17.637364 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:17.637400 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:36:17.638072 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:17.638088 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:17.638441 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:17.638797 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:17.639685 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:17.639690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:17.639702 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:36:17.639713 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:17.639767 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:36:17.639784 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:17.639937 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:17.640010 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 11:36:17.640258 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":161,"threads_started":1}
I20260926 11:36:17.643334 10198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3: Bootstrap starting.
I20260926 11:36:17.645846 10198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3: Neither blocks nor log segments found. Creating new log.
I20260926 11:36:17.646718 10198 log.cc:824] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3: Log is configured to *not* fsync() on all Append() calls
I20260926 11:36:17.649135 10198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3: No bootstrap required, opened a new log
I20260926 11:36:17.653548 10198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40079 } }
I20260926 11:36:17.653843 10198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:36:17.653936 10198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6716dd8cf5b34d4a8b1d5fd9bedf41d3, State: Initialized, Role: FOLLOWER
I20260926 11:36:17.654435 10198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40079 } }
I20260926 11:36:17.654592 10198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:36:17.654675 10198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:36:17.654781 10198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:36:17.655925 10198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40079 } }
I20260926 11:36:17.656353 10198 leader_election.cc:302] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6716dd8cf5b34d4a8b1d5fd9bedf41d3; no voters:
I20260926 11:36:17.656792 10198 leader_election.cc:288] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:36:17.656904 10203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:36:17.657150 10203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [term 1 LEADER]: Becoming Leader. State: Replica: 6716dd8cf5b34d4a8b1d5fd9bedf41d3, State: Running, Role: LEADER
I20260926 11:36:17.657552 10203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40079 } }
I20260926 11:36:17.658191 10198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:36:17.659396 10204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40079 } } }
I20260926 11:36:17.659528 10204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [sys.catalog]: This master's current role is: LEADER
I20260926 11:36:17.659801 10205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6716dd8cf5b34d4a8b1d5fd9bedf41d3. Latest consensus state: current_term: 1 leader_uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6716dd8cf5b34d4a8b1d5fd9bedf41d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40079 } } }
I20260926 11:36:17.659901 10205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3 [sys.catalog]: This master's current role is: LEADER
I20260926 11:36:17.660005 10212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:36:17.663599 10212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:36:17.670168 10212 catalog_manager.cc:1347] Generated new cluster ID: f8f82e1e44a94254b1ba1866846f37a0
I20260926 11:36:17.670264 10212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:36:17.684733 10212 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:36:17.685135 10212 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:36:17.690048 10212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6716dd8cf5b34d4a8b1d5fd9bedf41d3: Generated new TSK 0
I20260926 11:36:17.690802 10212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:36:17.701678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40079
--builtin_ntp_servers=127.0.98.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 11:36:17.831657 10222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:17.832005 10222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:17.832052 10222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:17.836408 10222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:17.836491 10222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:17.836555 10222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:36:17.841948 10222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:17.843097 10222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:17.844619 10222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:17.855852 10230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:17.855839 10228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:17.855839 10227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:17.857102 10222 server_base.cc:1034] running on GCE node
I20260926 11:36:17.857607 10222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:17.858371 10222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:17.859601 10222 hybrid_clock.cc:648] HybridClock initialized: now 1790422577859532 us; error 78 us; skew 500 ppm
I20260926 11:36:17.861708 10222 webserver.cc:460] Webserver started at http://127.0.98.1:37735/ using document root <none> and password file <none>
I20260926 11:36:17.862278 10222 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:17.862337 10222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:17.862531 10222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:17.864727 10222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a2ddfbd7e094a58a3bd4ef2cb3eca1c"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "3f0b55080125dccdc058e32cec745039"
server_key_iv: "6b974a9fd817280aefea02b791505f3d"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.865283 10222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a2ddfbd7e094a58a3bd4ef2cb3eca1c"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "3f0b55080125dccdc058e32cec745039"
server_key_iv: "6b974a9fd817280aefea02b791505f3d"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.869391 10222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:36:17.872278 10236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.873836 10222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:17.874007 10222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "0a2ddfbd7e094a58a3bd4ef2cb3eca1c"
format_stamp: "Formatted at 2026-09-26 11:36:17 on dist-test-slave-4pw5"
server_key: "3f0b55080125dccdc058e32cec745039"
server_key_iv: "6b974a9fd817280aefea02b791505f3d"
server_key_version: "encryptionkey@0"
I20260926 11:36:17.874161 10222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:17.879316 10222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:17.880069 10222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:17.880270 10222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:17.880884 10222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:17.881872 10222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:17.881945 10222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.882016 10222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:17.882061 10222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:17.892091 10222 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37765
I20260926 11:36:17.892134 10349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37765 every 8 connection(s)
I20260926 11:36:17.893105 10222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:36:17.899699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 10222
I20260926 11:36:17.899732 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:17.895122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59939 (local address 127.0.98.62:40079)
0926 11:36:17.895278 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:17.895283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:17.896515 (+ 1232us) server_negotiation.cc:407] Connection header received
0926 11:36:17.897826 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:17.897829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:17.897868 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:17.897949 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:17.899238 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:17.899241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:17.899243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:36:17.899256 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:17.899277 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:36:17.899286 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:17.899410 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:17.899499 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 11:36:17.899550 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:36:17.899820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:36:17.900127 392 external_mini_cluster.cc:1428] Setting key 15217f222b0ff6e7ea72c906c65e7a13
I20260926 11:36:17.900668 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:17.895675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40079 (local address 127.0.98.1:59939)
0926 11:36:17.896342 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:17.896382 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:17.897621 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:17.898116 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:17.898123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:17.898210 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:17.898959 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:36:17.898960 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:17.899000 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:36:17.899070 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:17.899372 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:17.899376 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:17.899380 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 11:36:17.899512 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 11:36:17.899786 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260926 11:36:17.902055 10350 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40079
I20260926 11:36:17.902292 10350 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:17.902333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40079
--builtin_ntp_servers=127.0.98.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 11:36:17.902801 10350 heartbeater.cc:511] Master 127.0.98.62:40079 requested a full tablet report, sending...
I20260926 11:36:17.904753 10161 ts_manager.cc:194] Registered new tserver with Master: 0a2ddfbd7e094a58a3bd4ef2cb3eca1c (127.0.98.1:37765)
W20260926 11:36:18.022476 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:18.022770 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:18.022819 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:18.027066 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:18.027136 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:18.027196 10353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:36:18.032295 10353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:18.033476 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:18.034863 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:18.045334 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:18.045342 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:18.045677 10353 server_base.cc:1034] running on GCE node
W20260926 11:36:18.045706 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:18.046214 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:18.046898 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:18.048086 10353 hybrid_clock.cc:648] HybridClock initialized: now 1790422578048048 us; error 56 us; skew 500 ppm
I20260926 11:36:18.050190 10353 webserver.cc:460] Webserver started at http://127.0.98.2:34369/ using document root <none> and password file <none>
I20260926 11:36:18.050815 10353 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:18.050920 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:18.051172 10353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:18.053428 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "736add3f41fe4c88a7fa6d94a39383f9"
format_stamp: "Formatted at 2026-09-26 11:36:18 on dist-test-slave-4pw5"
server_key: "58339da925f6936c78f58413ea438842"
server_key_iv: "9435912494f17e54dd1a9863cdb1f1c6"
server_key_version: "encryptionkey@0"
I20260926 11:36:18.054030 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "736add3f41fe4c88a7fa6d94a39383f9"
format_stamp: "Formatted at 2026-09-26 11:36:18 on dist-test-slave-4pw5"
server_key: "58339da925f6936c78f58413ea438842"
server_key_iv: "9435912494f17e54dd1a9863cdb1f1c6"
server_key_version: "encryptionkey@0"
I20260926 11:36:18.058014 10353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 11:36:18.061468 10367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:18.062914 10353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:36:18.063165 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "736add3f41fe4c88a7fa6d94a39383f9"
format_stamp: "Formatted at 2026-09-26 11:36:18 on dist-test-slave-4pw5"
server_key: "58339da925f6936c78f58413ea438842"
server_key_iv: "9435912494f17e54dd1a9863cdb1f1c6"
server_key_version: "encryptionkey@0"
I20260926 11:36:18.063311 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:18.068652 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:18.069404 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:18.069628 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:18.070253 10353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:18.071285 10353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:18.071362 10353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:18.071432 10353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:18.071475 10353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:18.081877 10353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42061
I20260926 11:36:18.081890 10480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42061 every 8 connection(s)
I20260926 11:36:18.082899 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:36:18.089359 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:18.084852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43763 (local address 127.0.98.62:40079)
0926 11:36:18.084997 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:18.085001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:18.086095 (+ 1094us) server_negotiation.cc:407] Connection header received
0926 11:36:18.087530 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:18.087534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:18.087627 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:18.087739 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:18.088845 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:18.088850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:18.088852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:36:18.088874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:18.088902 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:36:18.088911 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:18.089020 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:18.089154 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 11:36:18.089203 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:36:18.090087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 10353
I20260926 11:36:18.090230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:36:18.090349 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:18.085188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40079 (local address 127.0.98.2:43763)
0926 11:36:18.085936 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:18.085980 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:18.087232 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:18.087911 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:18.087918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:18.088007 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:18.088581 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:36:18.088582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:18.088621 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:36:18.088688 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:18.088999 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:18.089007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:18.089014 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:36:18.089201 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 11:36:18.089455 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":115,"threads_started":1}
I20260926 11:36:18.090603 392 external_mini_cluster.cc:1428] Setting key 7219b7830fdcb94652dfae39c069a268
I20260926 11:36:18.091728 10481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40079
I20260926 11:36:18.091987 10481 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:18.092487 10481 heartbeater.cc:511] Master 127.0.98.62:40079 requested a full tablet report, sending...
I20260926 11:36:18.092864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40079
--builtin_ntp_servers=127.0.98.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 11:36:18.093590 10161 ts_manager.cc:194] Registered new tserver with Master: 736add3f41fe4c88a7fa6d94a39383f9 (127.0.98.2:42061)
W20260926 11:36:18.214810 10484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:18.215154 10484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:18.215231 10484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:18.219705 10484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:18.219823 10484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:18.219920 10484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:36:18.225342 10484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:18.226686 10484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:18.228453 10484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:18.239480 10489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:18.239480 10492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:18.239480 10490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:18.239534 10484 server_base.cc:1034] running on GCE node
I20260926 11:36:18.240202 10484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:18.240949 10484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:18.242161 10484 hybrid_clock.cc:648] HybridClock initialized: now 1790422578242129 us; error 49 us; skew 500 ppm
I20260926 11:36:18.244377 10484 webserver.cc:460] Webserver started at http://127.0.98.3:33535/ using document root <none> and password file <none>
I20260926 11:36:18.244962 10484 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:18.245025 10484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:18.245199 10484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:18.247344 10484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "b95e29efac7e4d589d00be93a1b154bc"
format_stamp: "Formatted at 2026-09-26 11:36:18 on dist-test-slave-4pw5"
server_key: "7c69b079681ee99bfa2e7e8686aa18ed"
server_key_iv: "3d39565fa3d8be7518fd32386a00e8e1"
server_key_version: "encryptionkey@0"
I20260926 11:36:18.248095 10484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b95e29efac7e4d589d00be93a1b154bc"
format_stamp: "Formatted at 2026-09-26 11:36:18 on dist-test-slave-4pw5"
server_key: "7c69b079681ee99bfa2e7e8686aa18ed"
server_key_iv: "3d39565fa3d8be7518fd32386a00e8e1"
server_key_version: "encryptionkey@0"
I20260926 11:36:18.252300 10484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:36:18.255424 10498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:18.257026 10484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:18.257174 10484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "b95e29efac7e4d589d00be93a1b154bc"
format_stamp: "Formatted at 2026-09-26 11:36:18 on dist-test-slave-4pw5"
server_key: "7c69b079681ee99bfa2e7e8686aa18ed"
server_key_iv: "3d39565fa3d8be7518fd32386a00e8e1"
server_key_version: "encryptionkey@0"
I20260926 11:36:18.257284 10484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:18.262617 10484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:18.263290 10484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:18.263496 10484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:18.264176 10484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:18.265131 10484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:18.265178 10484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:18.265219 10484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:18.265269 10484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:18.275100 10484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33601
I20260926 11:36:18.275117 10611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33601 every 8 connection(s)
I20260926 11:36:18.276160 10484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:36:18.280251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 10484
I20260926 11:36:18.280385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:36:18.280735 392 external_mini_cluster.cc:1428] Setting key 56439a534234c3b1d00454acac8032c7
I20260926 11:36:18.283387 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:18.278067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51589 (local address 127.0.98.62:40079)
0926 11:36:18.278274 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:18.278278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:18.279467 (+ 1189us) server_negotiation.cc:407] Connection header received
0926 11:36:18.281268 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:18.281272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:18.281343 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:18.281451 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:18.282875 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:18.282880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:18.282883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:36:18.282898 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:18.282924 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:36:18.282934 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:18.283038 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:18.283238 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 11:36:18.283278 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 11:36:18.284502 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:18.278557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40079 (local address 127.0.98.3:51589)
0926 11:36:18.279305 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:18.279356 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:18.281017 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:18.281654 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:18.281661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:18.281779 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:18.282456 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:36:18.282458 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:18.282584 (+ 126us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:36:18.282698 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:18.283038 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:18.283051 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:18.283060 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 11:36:18.283272 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 11:36:18.283563 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260926 11:36:18.285746 10612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40079
I20260926 11:36:18.286015 10612 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:18.286474 10612 heartbeater.cc:511] Master 127.0.98.62:40079 requested a full tablet report, sending...
I20260926 11:36:18.287624 10161 ts_manager.cc:194] Registered new tserver with Master: b95e29efac7e4d589d00be93a1b154bc (127.0.98.3:33601)
I20260926 11:36:18.295819 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 11:36:18.299033 10201 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:36:18.297991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.0.98.62:40079)
0926 11:36:18.298185 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:18.298189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:18.298287 (+ 98us) server_negotiation.cc:407] Connection header received
0926 11:36:18.298445 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:18.298448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:18.298500 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:18.298664 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:18.298888 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59072: BlockingRecv error: recv got EOF from 127.0.0.1:59072 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 11:36:18.299505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 10222
I20260926 11:36:18.305909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 10353
I20260926 11:36:18.311165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 10484
I20260926 11:36:18.317044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 10131
2026-09-26T11:36:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (890 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:36:18 dist-test-slave-4pw5 krb5kdc[10622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:36:18 dist-test-slave-4pw5 krb5kdc[10622](info): set up 2 sockets
Sep 26 11:36:18 dist-test-slave-4pw5 krb5kdc[10622](info): commencing operation
krb5kdc: starting...
W20260926 11:36:19.456708 392 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:36:19 dist-test-slave-4pw5 krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:36:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:36:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:36:19.592444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:36:19.713234 10635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:19.713533 10635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:19.713577 10635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:19.717669 10635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:36:19.717743 10635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:19.717763 10635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:19.717782 10635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:36:19.717797 10635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:36:19.723315 10635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:19.724526 10635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:19.725569 10635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:19.734877 10641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:19.734804 10640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:19.734800 10643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:19.735001 10635 server_base.cc:1034] running on GCE node
I20260926 11:36:19.735857 10635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:19.737023 10635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:19.738193 10635 hybrid_clock.cc:648] HybridClock initialized: now 1790422579738168 us; error 42 us; skew 500 ppm
Sep 26 11:36:19 dist-test-slave-4pw5 krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:19.741339 10635 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:36:19.742389 10635 webserver.cc:460] Webserver started at http://127.0.98.62:36899/ using document root <none> and password file <none>
I20260926 11:36:19.742996 10635 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:19.743084 10635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:19.743321 10635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:19.745568 10635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "de7371c03dd544189d1cc0f69e2ed294"
format_stamp: "Formatted at 2026-09-26 11:36:19 on dist-test-slave-4pw5"
server_key: "0c1ec37bd3721804151462f89ec5a50b"
server_key_iv: "5a25955ca24558bfd920dec29b6fc83b"
server_key_version: "encryptionkey@0"
I20260926 11:36:19.746183 10635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "de7371c03dd544189d1cc0f69e2ed294"
format_stamp: "Formatted at 2026-09-26 11:36:19 on dist-test-slave-4pw5"
server_key: "0c1ec37bd3721804151462f89ec5a50b"
server_key_iv: "5a25955ca24558bfd920dec29b6fc83b"
server_key_version: "encryptionkey@0"
I20260926 11:36:19.750303 10635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 11:36:19.753273 10650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:19.754642 10635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:36:19.754802 10635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "de7371c03dd544189d1cc0f69e2ed294"
format_stamp: "Formatted at 2026-09-26 11:36:19 on dist-test-slave-4pw5"
server_key: "0c1ec37bd3721804151462f89ec5a50b"
server_key_iv: "5a25955ca24558bfd920dec29b6fc83b"
server_key_version: "encryptionkey@0"
I20260926 11:36:19.754951 10635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:19.809422 10635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:19.812999 10635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:19.813232 10635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:19.820871 10635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46681
I20260926 11:36:19.821216 10702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46681 every 8 connection(s)
I20260926 11:36:19.821939 10635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:36:19.824838 10703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:36:19.830401 10703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294: Bootstrap starting.
I20260926 11:36:19.830718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 10635
I20260926 11:36:19.830862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:36:19.831151 392 external_mini_cluster.cc:1428] Setting key 2634e951f958322e3f3e48d2b4ef8f21
I20260926 11:36:19.833190 10703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294: Neither blocks nor log segments found. Creating new log.
I20260926 11:36:19.833945 10703 log.cc:824] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294: Log is configured to *not* fsync() on all Append() calls
I20260926 11:36:19.836620 10703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294: No bootstrap required, opened a new log
Sep 26 11:36:19 dist-test-slave-4pw5 krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:36:19.841993 10703 raft_consensus.cc:348] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7371c03dd544189d1cc0f69e2ed294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } }
I20260926 11:36:19.842341 10703 raft_consensus.cc:374] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:36:19.842438 10703 raft_consensus.cc:729] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7371c03dd544189d1cc0f69e2ed294, State: Initialized, Role: FOLLOWER
I20260926 11:36:19.843040 10703 consensus_queue.cc:260] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7371c03dd544189d1cc0f69e2ed294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } }
I20260926 11:36:19.843258 10703 raft_consensus.cc:388] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:36:19.843355 10703 raft_consensus.cc:482] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:36:19.843459 10703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:36:19.845026 10703 raft_consensus.cc:504] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7371c03dd544189d1cc0f69e2ed294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } }
I20260926 11:36:19.845492 10703 leader_election.cc:302] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de7371c03dd544189d1cc0f69e2ed294; no voters:
I20260926 11:36:19.845927 10703 leader_election.cc:288] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:36:19.846040 10708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:36:19.846287 10708 raft_consensus.cc:686] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [term 1 LEADER]: Becoming Leader. State: Replica: de7371c03dd544189d1cc0f69e2ed294, State: Running, Role: LEADER
I20260926 11:36:19.846660 10708 consensus_queue.cc:237] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7371c03dd544189d1cc0f69e2ed294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } }
I20260926 11:36:19.847510 10703 sys_catalog.cc:564] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:36:19.848376 10709 sys_catalog.cc:455] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de7371c03dd544189d1cc0f69e2ed294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7371c03dd544189d1cc0f69e2ed294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } } }
I20260926 11:36:19.848516 10709 sys_catalog.cc:458] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [sys.catalog]: This master's current role is: LEADER
I20260926 11:36:19.848949 10710 sys_catalog.cc:455] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de7371c03dd544189d1cc0f69e2ed294. Latest consensus state: current_term: 1 leader_uuid: "de7371c03dd544189d1cc0f69e2ed294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7371c03dd544189d1cc0f69e2ed294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } } }
I20260926 11:36:19.849083 10710 sys_catalog.cc:458] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294 [sys.catalog]: This master's current role is: LEADER
I20260926 11:36:19.849361 10717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:36:19.849748 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:19.832757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.0.98.62:46681)
0926 11:36:19.833207 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:19.833224 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:19.833260 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:36:19.834066 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:19.834128 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:19.834576 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:19.834955 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:19.836236 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:19.837361 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:19.838776 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:19.839302 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:19.839427 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:19.842891 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:19.842900 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:19.842933 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:36:19.842978 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:19.845454 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:19.845994 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:19.846001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:19.846009 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:19.846186 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:19.846730 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:19.846733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:19.846734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:19.847278 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:19.847515 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:19.848222 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 11:36:19.848568 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":157,"threads_started":1}
I20260926 11:36:19.852818 10717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:36:19.859081 10717 catalog_manager.cc:1347] Generated new cluster ID: 7d8a7154bd7d4ff598859d7842a2c501
I20260926 11:36:19.859171 10717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:36:19.874370 10717 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:36:19.875754 10717 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:36:19.892211 10717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294: Generated new TSK 0
I20260926 11:36:19.893075 10717 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:36:19.928471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:36:20.076603 10729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:20.076946 10729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:20.077026 10729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:20.081239 10729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:20.081331 10729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:20.081418 10729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:36:20.087090 10729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:20.088294 10729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:20.089330 10729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:20.099227 10735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:20.099200 10734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:20.099200 10737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:20.099239 10729 server_base.cc:1034] running on GCE node
I20260926 11:36:20.099967 10729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:20.100646 10729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:20.101830 10729 hybrid_clock.cc:648] HybridClock initialized: now 1790422580101807 us; error 38 us; skew 500 ppm
Sep 26 11:36:20 dist-test-slave-4pw5 krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:20.107874 10729 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:36:20.108909 10729 webserver.cc:460] Webserver started at http://127.0.98.1:43607/ using document root <none> and password file <none>
I20260926 11:36:20.109505 10729 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:20.109596 10729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:20.109817 10729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:20.111992 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce67d7a79c98402f9068a221dcda0a2b"
format_stamp: "Formatted at 2026-09-26 11:36:20 on dist-test-slave-4pw5"
server_key: "b7b2b8d72a57a0c28e6d165cbf337856"
server_key_iv: "97bfbd09a2c2c4687ce310179f2493b1"
server_key_version: "encryptionkey@0"
I20260926 11:36:20.112560 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce67d7a79c98402f9068a221dcda0a2b"
format_stamp: "Formatted at 2026-09-26 11:36:20 on dist-test-slave-4pw5"
server_key: "b7b2b8d72a57a0c28e6d165cbf337856"
server_key_iv: "97bfbd09a2c2c4687ce310179f2493b1"
server_key_version: "encryptionkey@0"
I20260926 11:36:20.116556 10729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:36:20.119292 10744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:20.120846 10729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:20.121016 10729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "ce67d7a79c98402f9068a221dcda0a2b"
format_stamp: "Formatted at 2026-09-26 11:36:20 on dist-test-slave-4pw5"
server_key: "b7b2b8d72a57a0c28e6d165cbf337856"
server_key_iv: "97bfbd09a2c2c4687ce310179f2493b1"
server_key_version: "encryptionkey@0"
I20260926 11:36:20.121142 10729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:20.135396 10729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:20.139833 10729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:20.140053 10729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:20.140671 10729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:20.141511 10729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:20.141556 10729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:20.141595 10729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:20.141645 10729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:20.150630 10729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43403
I20260926 11:36:20.150592 10857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43403 every 8 connection(s)
I20260926 11:36:20.151552 10729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:36:20.156718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 10729
I20260926 11:36:20.156833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:36:20.157155 392 external_mini_cluster.cc:1428] Setting key 9d9892fd007d8ae8a4473c769519527c
Sep 26 11:36:20 dist-test-slave-4pw5 krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:36:20.166857 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.153563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35521 (local address 127.0.98.62:46681)
0926 11:36:20.153720 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.153726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.154773 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 11:36:20.155706 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.155709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.155755 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.155830 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:20.157317 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.158019 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.159481 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.159973 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.160107 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.163011 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:20.163015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:20.163017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:36:20.163042 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:20.164706 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:20.165249 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:20.165253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:20.165255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:20.165339 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:20.165700 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:20.165704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:20.165705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:20.165903 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:20.166013 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.166546 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.166685 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:36:20.167761 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.153954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46681 (local address 127.0.98.1:35521)
0926 11:36:20.154625 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:20.154667 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:20.155449 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:20.156030 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:20.156036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:20.156405 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:20.157143 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.157164 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.158145 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.158149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.159294 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.159305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.160125 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.160128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.160281 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.161050 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:20.161077 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:20.162821 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:20.164835 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:20.164841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:20.164860 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:20.165143 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:20.165429 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:20.165434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:20.165436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:20.165598 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:20.166014 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:20.166022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:20.166299 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 11:36:20.166505 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 11:36:20.166852 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":114,"threads_started":1}
I20260926 11:36:20.169126 10858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46681
I20260926 11:36:20.169486 10858 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:36:20.170050 10858 heartbeater.cc:511] Master 127.0.98.62:46681 requested a full tablet report, sending...
I20260926 11:36:20.172026 10667 ts_manager.cc:194] Registered new tserver with Master: ce67d7a79c98402f9068a221dcda0a2b (127.0.98.1:43403)
I20260926 11:36:20.173393 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:36:20.190518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:36:20.308946 10863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:20.309331 10863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:20.309422 10863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:20.313585 10863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:20.313695 10863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:20.313827 10863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:36:20.319536 10863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:20.320729 10863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:20.321754 10863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:20.332021 10868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:20.332109 10871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:20.331940 10869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:20.332341 10863 server_base.cc:1034] running on GCE node
I20260926 11:36:20.332760 10863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:20.333439 10863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:20.334630 10863 hybrid_clock.cc:648] HybridClock initialized: now 1790422580334605 us; error 36 us; skew 500 ppm
Sep 26 11:36:20 dist-test-slave-4pw5 krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:20.337811 10863 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:36:20.338792 10863 webserver.cc:460] Webserver started at http://127.0.98.2:37439/ using document root <none> and password file <none>
I20260926 11:36:20.339378 10863 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:20.339478 10863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:20.339756 10863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:20.341909 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "77f440c1aabd4a10964a7bbb6155e8c7"
format_stamp: "Formatted at 2026-09-26 11:36:20 on dist-test-slave-4pw5"
server_key: "bd86f82553025cbad4496a520b9822fc"
server_key_iv: "90b203c775fa6018d3a98b0571cbcdc0"
server_key_version: "encryptionkey@0"
I20260926 11:36:20.342525 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "77f440c1aabd4a10964a7bbb6155e8c7"
format_stamp: "Formatted at 2026-09-26 11:36:20 on dist-test-slave-4pw5"
server_key: "bd86f82553025cbad4496a520b9822fc"
server_key_iv: "90b203c775fa6018d3a98b0571cbcdc0"
server_key_version: "encryptionkey@0"
I20260926 11:36:20.346462 10863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 11:36:20.349186 10878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:20.350745 10863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:36:20.350899 10863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "77f440c1aabd4a10964a7bbb6155e8c7"
format_stamp: "Formatted at 2026-09-26 11:36:20 on dist-test-slave-4pw5"
server_key: "bd86f82553025cbad4496a520b9822fc"
server_key_iv: "90b203c775fa6018d3a98b0571cbcdc0"
server_key_version: "encryptionkey@0"
I20260926 11:36:20.351034 10863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:20.363988 10863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:20.367509 10863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:20.367764 10863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:20.368395 10863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:20.369270 10863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:20.369343 10863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:20.369431 10863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:20.369488 10863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:20.379843 10863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44107
I20260926 11:36:20.379863 10991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44107 every 8 connection(s)
I20260926 11:36:20.380808 10863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:36:20.387713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 10863
I20260926 11:36:20.387842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:36:20.388121 392 external_mini_cluster.cc:1428] Setting key 97acd20f79287690fe63407821b208d6
Sep 26 11:36:20 dist-test-slave-4pw5 krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:36:20.395879 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.382751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59783 (local address 127.0.98.62:46681)
0926 11:36:20.382885 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.382889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.384073 (+ 1184us) server_negotiation.cc:407] Connection header received
0926 11:36:20.385004 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.385007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.385067 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.385173 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:20.386365 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.386913 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.388157 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.388427 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.388545 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.391675 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:20.391680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:20.391685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:36:20.391717 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:20.393575 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:20.394188 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:20.394192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:20.394194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:20.394281 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:20.394643 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:20.394647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:20.394649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:20.394856 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:20.394966 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.395548 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.395699 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:36:20.396730 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.383084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46681 (local address 127.0.98.2:59783)
0926 11:36:20.383915 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:20.383957 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:20.384785 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:20.385344 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:20.385351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:20.385735 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:20.386210 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.386224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.387052 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.387055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.388034 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.388045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.388592 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.388595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.388825 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.389603 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:20.389620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:20.391410 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:20.393708 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:20.393713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:20.393728 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:20.394046 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:20.394376 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:20.394388 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:20.394390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:20.394541 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:20.394965 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:20.394972 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:20.395241 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 11:36:20.395506 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 11:36:20.395888 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:36:20.398058 10992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46681
I20260926 11:36:20.398335 10992 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:20.398958 10992 heartbeater.cc:511] Master 127.0.98.62:46681 requested a full tablet report, sending...
I20260926 11:36:20.400197 10667 ts_manager.cc:194] Registered new tserver with Master: 77f440c1aabd4a10964a7bbb6155e8c7 (127.0.98.2:44107)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:36:20.401017 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:36:20.418588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:36:20.535856 10997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:36:20.536151 10997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:36:20.536197 10997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:36:20.540489 10997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:36:20.540565 10997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:36:20.540652 10997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:36:20.546296 10997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:36:20.547459 10997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:36:20.548653 10997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:36:20.559124 11005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:20.559104 11003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:36:20.559085 11002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:36:20.559345 10997 server_base.cc:1034] running on GCE node
I20260926 11:36:20.559903 10997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:36:20.560672 10997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:36:20.561865 10997 hybrid_clock.cc:648] HybridClock initialized: now 1790422580561831 us; error 52 us; skew 500 ppm
Sep 26 11:36:20 dist-test-slave-4pw5 krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:36:20.565042 10997 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:36:20.566053 10997 webserver.cc:460] Webserver started at http://127.0.98.3:45565/ using document root <none> and password file <none>
I20260926 11:36:20.566680 10997 fs_manager.cc:359] Metadata directory not provided
I20260926 11:36:20.566771 10997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:36:20.567004 10997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:36:20.569166 10997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "51afa53e90254181801e917bc2f19201"
format_stamp: "Formatted at 2026-09-26 11:36:20 on dist-test-slave-4pw5"
server_key: "dfa673efc196036a722e74f9c078602b"
server_key_iv: "7ae14300dc28e84eceab80110139cb07"
server_key_version: "encryptionkey@0"
I20260926 11:36:20.569742 10997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51afa53e90254181801e917bc2f19201"
format_stamp: "Formatted at 2026-09-26 11:36:20 on dist-test-slave-4pw5"
server_key: "dfa673efc196036a722e74f9c078602b"
server_key_iv: "7ae14300dc28e84eceab80110139cb07"
server_key_version: "encryptionkey@0"
I20260926 11:36:20.573736 10997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:36:20.576756 11012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:20.578222 10997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:36:20.578387 10997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "51afa53e90254181801e917bc2f19201"
format_stamp: "Formatted at 2026-09-26 11:36:20 on dist-test-slave-4pw5"
server_key: "dfa673efc196036a722e74f9c078602b"
server_key_iv: "7ae14300dc28e84eceab80110139cb07"
server_key_version: "encryptionkey@0"
I20260926 11:36:20.578536 10997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:36:20.592842 10997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:36:20.596619 10997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:36:20.596850 10997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:36:20.597465 10997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:36:20.598315 10997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:36:20.598383 10997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:20.598461 10997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:36:20.598506 10997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:36:20.607790 10997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36905
I20260926 11:36:20.607820 11125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36905 every 8 connection(s)
I20260926 11:36:20.608768 10997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:36:20.616029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 10997
I20260926 11:36:20.616190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:36:20.616532 392 external_mini_cluster.cc:1428] Setting key f58c59c5ebbc294058045ed3ea524a01
Sep 26 11:36:20 dist-test-slave-4pw5 krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:36:20.623687 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.610767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33963 (local address 127.0.98.62:46681)
0926 11:36:20.610932 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.610938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.612194 (+ 1256us) server_negotiation.cc:407] Connection header received
0926 11:36:20.613103 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.613108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.613159 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.613242 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:20.614468 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.615023 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.616420 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.616799 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.616925 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.619702 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:20.619706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:20.619708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:36:20.619733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:20.621218 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:20.621814 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:20.621817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:20.621818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:20.621896 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:20.622258 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:20.622260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:20.622261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:20.622450 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:20.622530 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.623124 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.623485 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:36:20.624308 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.611237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46681 (local address 127.0.98.3:33963)
0926 11:36:20.612029 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:20.612070 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:20.612882 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:20.613400 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:20.613408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:20.613817 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:36:20.614300 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.614313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.615155 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.615158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.616296 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.616306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.616973 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.616978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.617181 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.617855 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:36:20.617872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:36:20.619482 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:36:20.621342 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:20.621347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:20.621362 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:20.621702 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:20.622004 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:36:20.622006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:36:20.622008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:36:20.622158 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:36:20.622588 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:36:20.622593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:36:20.622853 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 11:36:20.623022 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 11:36:20.623306 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":128,"threads_started":1}
I20260926 11:36:20.625665 11126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46681
I20260926 11:36:20.625962 11126 heartbeater.cc:464] Registering TS with master...
I20260926 11:36:20.626576 11126 heartbeater.cc:511] Master 127.0.98.62:46681 requested a full tablet report, sending...
I20260926 11:36:20.627872 10667 ts_manager.cc:194] Registered new tserver with Master: 51afa53e90254181801e917bc2f19201 (127.0.98.3:36905)
I20260926 11:36:20.628532 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33963
I20260926 11:36:20.630466 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:36:20.640357 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.632591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.0.98.62:46681)
0926 11:36:20.632762 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.632768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.632876 (+ 108us) server_negotiation.cc:407] Connection header received
0926 11:36:20.632982 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.632984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.633024 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.633105 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:36:20.633855 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.634380 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.635196 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.635409 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.635491 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.636965 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:36:20.636968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:36:20.636970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:36:20.636994 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:36:20.638376 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:20.638852 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:20.638854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:20.638856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:20.639007 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:36:20.639331 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:36:20.639334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:36:20.639335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:36:20.639666 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:36:20.639783 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.640020 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.640153 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 11:36:20.643790 10667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:36:20.646813 10667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:36:20.663587 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.656467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44107 (local address 127.0.0.1:33724)
0926 11:36:20.656917 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:20.656990 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:20.657119 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:20.658565 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:20.658571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:20.658612 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:36:20.658749 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.658755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.660569 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.660573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.662016 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.662027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.663040 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.663044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.663140 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.663161 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:36:20.663212 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 11:36:20.663354 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":129,"thread_start_us":134,"threads_started":1}
I20260926 11:36:20.663666 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.657706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.0.98.3:36905)
0926 11:36:20.658827 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.658839 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.658859 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:36:20.658930 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.658937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.659097 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.659299 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:36:20.659487 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.660256 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.661942 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.662759 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.662875 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.663056 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.663379 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.663483 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":151,"threads_started":1}
I20260926 11:36:20.663802 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.657317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.0.98.2:44107)
0926 11:36:20.658138 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.658152 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.658174 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:36:20.658240 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.658244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.658382 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.658523 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:36:20.658897 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.660289 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.662137 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.662896 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.663305 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.663453 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.663527 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.663654 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":160,"threads_started":1}
I20260926 11:36:20.663971 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.657484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36905 (local address 127.0.0.1:59804)
0926 11:36:20.657908 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:20.657921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:20.658094 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:20.659269 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:20.659272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:20.659289 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:36:20.659390 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.659396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.660407 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.660412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.661811 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.661821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.662893 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.662896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.663014 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.663052 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 11:36:20.663165 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 11:36:20.663839 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":531,"thread_start_us":74,"threads_started":1}
I20260926 11:36:20.664024 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.656720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.0.98.1:43403)
0926 11:36:20.657396 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.657414 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.657843 (+ 429us) server_negotiation.cc:407] Connection header received
0926 11:36:20.658115 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.658121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.658356 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.658498 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:36:20.658987 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.660257 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.662297 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.662888 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.662995 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.663144 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.663766 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.663876 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":96,"threads_started":1}
I20260926 11:36:20.664181 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.656467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43403 (local address 127.0.0.1:53172)
0926 11:36:20.657746 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:20.657761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:20.657917 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:20.658725 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:20.658727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:20.658745 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:36:20.658860 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.658867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.660419 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.660424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.662181 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.662188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.663123 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.663126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.663218 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.663236 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 11:36:20.663298 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 11:36:20.663383 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":85,"thread_start_us":116,"threads_started":1}
I20260926 11:36:20.666946 10926 tablet_service.cc:1467] Processing CreateTablet for tablet 51f5400a8a0c491abfc6d8e39c696415 (DEFAULT_TABLE table=test-table [id=4520517ea6f84408a7650905c0a9eec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:36:20.667048 11060 tablet_service.cc:1467] Processing CreateTablet for tablet 51f5400a8a0c491abfc6d8e39c696415 (DEFAULT_TABLE table=test-table [id=4520517ea6f84408a7650905c0a9eec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:36:20.667061 10792 tablet_service.cc:1467] Processing CreateTablet for tablet 51f5400a8a0c491abfc6d8e39c696415 (DEFAULT_TABLE table=test-table [id=4520517ea6f84408a7650905c0a9eec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:36:20.668010 10926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5400a8a0c491abfc6d8e39c696415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:36:20.668097 11060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5400a8a0c491abfc6d8e39c696415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:36:20.668092 10792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5400a8a0c491abfc6d8e39c696415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:36:20.674324 11143 tablet_bootstrap.cc:492] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: Bootstrap starting.
I20260926 11:36:20.675493 11144 tablet_bootstrap.cc:492] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b: Bootstrap starting.
I20260926 11:36:20.676173 11145 tablet_bootstrap.cc:492] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: Bootstrap starting.
I20260926 11:36:20.677501 11144 tablet_bootstrap.cc:654] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b: Neither blocks nor log segments found. Creating new log.
I20260926 11:36:20.677522 11143 tablet_bootstrap.cc:654] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: Neither blocks nor log segments found. Creating new log.
I20260926 11:36:20.678243 11144 log.cc:824] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b: Log is configured to *not* fsync() on all Append() calls
I20260926 11:36:20.678339 11143 log.cc:824] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: Log is configured to *not* fsync() on all Append() calls
I20260926 11:36:20.678741 11145 tablet_bootstrap.cc:654] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: Neither blocks nor log segments found. Creating new log.
I20260926 11:36:20.679390 11145 log.cc:824] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: Log is configured to *not* fsync() on all Append() calls
I20260926 11:36:20.680336 11144 tablet_bootstrap.cc:492] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b: No bootstrap required, opened a new log
I20260926 11:36:20.680559 11144 ts_tablet_manager.cc:1397] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:36:20.680820 11143 tablet_bootstrap.cc:492] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: No bootstrap required, opened a new log
I20260926 11:36:20.680990 11143 ts_tablet_manager.cc:1397] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 11:36:20.681224 11145 tablet_bootstrap.cc:492] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: No bootstrap required, opened a new log
I20260926 11:36:20.681375 11145 ts_tablet_manager.cc:1397] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:36:20.684624 11144 raft_consensus.cc:348] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } }
I20260926 11:36:20.684898 11144 raft_consensus.cc:374] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:36:20.684985 11144 raft_consensus.cc:729] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce67d7a79c98402f9068a221dcda0a2b, State: Initialized, Role: FOLLOWER
I20260926 11:36:20.684929 11143 raft_consensus.cc:348] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } }
I20260926 11:36:20.685233 11143 raft_consensus.cc:374] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:36:20.685281 11143 raft_consensus.cc:729] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f440c1aabd4a10964a7bbb6155e8c7, State: Initialized, Role: FOLLOWER
I20260926 11:36:20.685123 11145 raft_consensus.cc:348] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } }
I20260926 11:36:20.685355 11145 raft_consensus.cc:374] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:36:20.685412 11145 raft_consensus.cc:729] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51afa53e90254181801e917bc2f19201, State: Initialized, Role: FOLLOWER
I20260926 11:36:20.685504 11144 consensus_queue.cc:260] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } }
I20260926 11:36:20.685750 11143 consensus_queue.cc:260] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } }
I20260926 11:36:20.685778 11145 consensus_queue.cc:260] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } }
I20260926 11:36:20.686213 10858 heartbeater.cc:503] Master 127.0.98.62:46681 was elected leader, sending a full tablet report...
I20260926 11:36:20.686607 11144 ts_tablet_manager.cc:1428] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260926 11:36:20.686685 11145 ts_tablet_manager.cc:1428] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:36:20.686685 11143 ts_tablet_manager.cc:1428] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:36:20.687004 10992 heartbeater.cc:503] Master 127.0.98.62:46681 was elected leader, sending a full tablet report...
I20260926 11:36:20.687004 11126 heartbeater.cc:503] Master 127.0.98.62:46681 was elected leader, sending a full tablet report...
W20260926 11:36:20.859866 11127 tablet.cc:2369] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:36:20.882177 10993 tablet.cc:2369] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:36:20.903065 10859 tablet.cc:2369] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:36:20.913725 11150 raft_consensus.cc:482] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:36:20.913971 11150 raft_consensus.cc:504] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } }
I20260926 11:36:20.915735 11150 leader_election.cc:288] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce67d7a79c98402f9068a221dcda0a2b (127.0.98.1:43403), 77f440c1aabd4a10964a7bbb6155e8c7 (127.0.98.2:44107)
I20260926 11:36:20.921188 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.916107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54993 (local address 127.0.98.2:44107)
0926 11:36:20.916287 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.916292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.916317 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:36:20.916511 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.916514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.916590 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.916680 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:36:20.917106 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.917938 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.919822 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.920458 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.920594 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.920666 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.920981 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.921052 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 11:36:20.921187 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.916116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38897 (local address 127.0.98.1:43403)
0926 11:36:20.916247 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.916253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.916595 (+ 342us) server_negotiation.cc:407] Connection header received
0926 11:36:20.916659 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.916665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.916716 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.916791 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:36:20.917105 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.917916 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.919822 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.920513 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.920613 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.920665 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.920962 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.921048 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:36:20.921200 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.916009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44107 (local address 127.0.98.3:54993)
0926 11:36:20.916247 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:20.916263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:20.916373 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:20.916730 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:20.916733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:20.916750 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:36:20.916946 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.916953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.918081 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.918085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.919646 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.919661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.920621 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.920625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.920796 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.920843 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 11:36:20.920925 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 11:36:20.921027 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":241}
I20260926 11:36:20.921206 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.916009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43403 (local address 127.0.98.3:38897)
0926 11:36:20.916476 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:20.916490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:36:20.916575 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:36:20.916814 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:36:20.916817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:36:20.916829 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:36:20.916947 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.916953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.918081 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.918085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.919646 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:36:20.919661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.920621 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.920625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:36:20.920788 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.920838 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 11:36:20.920924 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 11:36:20.921029 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":295,"thread_start_us":143,"threads_started":1}
I20260926 11:36:20.921996 10946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5400a8a0c491abfc6d8e39c696415" candidate_uuid: "51afa53e90254181801e917bc2f19201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" is_pre_election: true
I20260926 11:36:20.922024 10812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5400a8a0c491abfc6d8e39c696415" candidate_uuid: "51afa53e90254181801e917bc2f19201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce67d7a79c98402f9068a221dcda0a2b" is_pre_election: true
I20260926 11:36:20.922314 10812 raft_consensus.cc:2454] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51afa53e90254181801e917bc2f19201 in term 0.
I20260926 11:36:20.922317 10946 raft_consensus.cc:2454] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51afa53e90254181801e917bc2f19201 in term 0.
I20260926 11:36:20.922887 11015 leader_election.cc:302] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51afa53e90254181801e917bc2f19201, 77f440c1aabd4a10964a7bbb6155e8c7, ce67d7a79c98402f9068a221dcda0a2b; no voters:
I20260926 11:36:20.923249 11150 raft_consensus.cc:2781] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:36:20.923347 11150 raft_consensus.cc:482] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:36:20.923409 11150 raft_consensus.cc:3037] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:36:20.924594 11150 raft_consensus.cc:504] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } }
I20260926 11:36:20.925033 11150 leader_election.cc:288] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [CANDIDATE]: Term 1 election: Requested vote from peers ce67d7a79c98402f9068a221dcda0a2b (127.0.98.1:43403), 77f440c1aabd4a10964a7bbb6155e8c7 (127.0.98.2:44107)
I20260926 11:36:20.925442 10946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5400a8a0c491abfc6d8e39c696415" candidate_uuid: "51afa53e90254181801e917bc2f19201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77f440c1aabd4a10964a7bbb6155e8c7"
I20260926 11:36:20.925452 10812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5400a8a0c491abfc6d8e39c696415" candidate_uuid: "51afa53e90254181801e917bc2f19201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce67d7a79c98402f9068a221dcda0a2b"
I20260926 11:36:20.925588 10946 raft_consensus.cc:3037] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:36:20.925588 10812 raft_consensus.cc:3037] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:36:20.926923 10812 raft_consensus.cc:2454] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51afa53e90254181801e917bc2f19201 in term 1.
I20260926 11:36:20.926923 10946 raft_consensus.cc:2454] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51afa53e90254181801e917bc2f19201 in term 1.
I20260926 11:36:20.927253 11013 leader_election.cc:302] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51afa53e90254181801e917bc2f19201, 77f440c1aabd4a10964a7bbb6155e8c7; no voters:
I20260926 11:36:20.927453 11150 raft_consensus.cc:2781] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:36:20.927739 11150 raft_consensus.cc:686] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 1 LEADER]: Becoming Leader. State: Replica: 51afa53e90254181801e917bc2f19201, State: Running, Role: LEADER
I20260926 11:36:20.928114 11150 consensus_queue.cc:237] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } }
I20260926 11:36:20.931721 10665 catalog_manager.cc:5495] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51afa53e90254181801e917bc2f19201 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "51afa53e90254181801e917bc2f19201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51afa53e90254181801e917bc2f19201" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36905 } health_report { overall_health: HEALTHY } } }
I20260926 11:36:20.967551 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:20.963494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.0.98.3:36905)
0926 11:36:20.963755 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 11:36:20.963759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:36:20.963773 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:36:20.963858 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:36:20.963860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:36:20.963919 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:36:20.964034 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:36:20.964309 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.964894 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.965774 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:36:20.966031 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:36:20.966126 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:36:20.966417 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:36:20.966922 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:36:20.967025 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:36:20.967190 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 11:36:20.967287 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260926 11:36:20.974452 10812 raft_consensus.cc:1261] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b [term 1 FOLLOWER]: Refusing update from remote peer 51afa53e90254181801e917bc2f19201: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:36:20.974447 10946 raft_consensus.cc:1261] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [term 1 FOLLOWER]: Refusing update from remote peer 51afa53e90254181801e917bc2f19201: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:36:20.975323 11150 consensus_queue.cc:1035] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77f440c1aabd4a10964a7bbb6155e8c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:36:20.975628 11153 consensus_queue.cc:1035] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce67d7a79c98402f9068a221dcda0a2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:36:20.984215 11158 mvcc.cc:205] Tried to move back new op lower bound from 7333570891661803520 to 7333570891484504064. Current Snapshot: MvccSnapshot[applied={T|T < 7333570891661803520}]
I20260926 11:36:20.985906 11157 mvcc.cc:205] Tried to move back new op lower bound from 7333570891661803520 to 7333570891484504064. Current Snapshot: MvccSnapshot[applied={T|T < 7333570891661803520}]
I20260926 11:36:20.994251 10667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:36:20.994611 10667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:36:20.997644 10667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de7371c03dd544189d1cc0f69e2ed294: Sending DeleteTablet for 3 replicas of tablet 51f5400a8a0c491abfc6d8e39c696415
I20260926 11:36:20.998649 10926 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f5400a8a0c491abfc6d8e39c696415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:36:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33724
I20260926 11:36:20.998967 10792 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f5400a8a0c491abfc6d8e39c696415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:36:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53172
I20260926 11:36:20.999027 11060 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f5400a8a0c491abfc6d8e39c696415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:36:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59804
I20260926 11:36:20.999514 11166 tablet_replica.cc:330] T 51f5400a8a0c491abfc6d8e39c696415 P ce67d7a79c98402f9068a221dcda0a2b: stopping tablet replica
I20260926 11:36:20.999819 11165 tablet_replica.cc:330] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: stopping tablet replica
I20260926 11:36:20.999876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 10729
I20260926 11:36:21.000586 11165 raft_consensus.cc:2229] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:36:21.000893 11167 tablet_replica.cc:330] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: stopping tablet replica
I20260926 11:36:21.000967 11165 raft_consensus.cc:2258] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:36:21.001209 11167 raft_consensus.cc:2229] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:36:21.001600 11167 raft_consensus.cc:2258] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:36:21.003628 11165 ts_tablet_manager.cc:1905] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:36:21.004200 11167 ts_tablet_manager.cc:1905] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:36:21.006786 11165 ts_tablet_manager.cc:1918] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:36:21.007025 11165 log.cc:1196] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/51f5400a8a0c491abfc6d8e39c696415
I20260926 11:36:21.007496 11165 ts_tablet_manager.cc:1939] T 51f5400a8a0c491abfc6d8e39c696415 P 77f440c1aabd4a10964a7bbb6155e8c7: Deleting consensus metadata
I20260926 11:36:21.007864 11167 ts_tablet_manager.cc:1918] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:36:21.008026 11167 log.cc:1196] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/51f5400a8a0c491abfc6d8e39c696415
I20260926 11:36:21.008432 11167 ts_tablet_manager.cc:1939] T 51f5400a8a0c491abfc6d8e39c696415 P 51afa53e90254181801e917bc2f19201: Deleting consensus metadata
I20260926 11:36:21.009316 10651 catalog_manager.cc:4841] TS 77f440c1aabd4a10964a7bbb6155e8c7 (127.0.98.2:44107): tablet 51f5400a8a0c491abfc6d8e39c696415 (table test-table [id=4520517ea6f84408a7650905c0a9eec8]) successfully deleted
I20260926 11:36:21.009646 10652 catalog_manager.cc:4841] TS 51afa53e90254181801e917bc2f19201 (127.0.98.3:36905): tablet 51f5400a8a0c491abfc6d8e39c696415 (table test-table [id=4520517ea6f84408a7650905c0a9eec8]) successfully deleted
W20260926 11:36:21.009902 10653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43403 (error 108)
I20260926 11:36:21.010708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 10863
I20260926 11:36:21.011020 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:36:21.010705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43403 (local address 127.0.0.1:53184)
0926 11:36:21.010838 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 11:36:21.010919 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 11:36:21.011432 10653 catalog_manager.cc:4584] TS ce67d7a79c98402f9068a221dcda0a2b (127.0.98.1:43403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51f5400a8a0c491abfc6d8e39c696415: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43403: connect: Connection refused (error 111)
I20260926 11:36:21.016949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 10997
I20260926 11:36:21.023401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 10635
2026-09-26T11:36:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2708 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:36:21 dist-test-slave-4pw5 krb5kdc[11171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:36:21 dist-test-slave-4pw5 krb5kdc[11171](info): set up 2 sockets
Sep 26 11:36:21 dist-test-slave-4pw5 krb5kdc[11171](info): commencing operation
krb5kdc: starting...
W20260926 11:36:22.092903 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:36:22 dist-test-slave-4pw5 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:36:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:36:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 11:36:22.283936 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPt_vxu/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/postgres -l logfile start
2026-09-26 11:36:25.173 UTC [11201] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 11:36:25.174 UTC [11201] LOG: listening on IPv4 address "127.0.98.20", port 57501
2026-09-26 11:36:25.178 UTC [11201] LOG: listening on Unix socket "/tmp/.s.PGSQL.57501"
2026-09-26 11:36:25.203 UTC [11204] LOG: database system was shut down at 2026-09-26 11:36:23 UTC
2026-09-26 11:36:25.207 UTC [11201] LOG: database system is ready to accept connections
I20260926 11:36:26.208403 392 mini_postgres.cc:96] Postgres bound to 57501
2026-09-26 11:36:26.215 UTC [11212] FATAL: database "slave" does not exist
127.0.98.20:57501 - accepting connections
I20260926 11:36:26.216284 392 mini_ranger.cc:161] Starting Ranger...
I20260926 11:36:26.233407 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 11:36:26.878816 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 11:36:26.878975 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-admin
2026-09-26 11:36:27,206 [I] DB FLAVOR :POSTGRES
2026-09-26 11:36:27,206 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 11:36:27,207 [I] Checking connection..
2026-09-26 11:36:27,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 11:36:27,509 [I] Checking connection passed.
2026-09-26 11:36:27,509 [I] --------- Verifying version history table ---------
2026-09-26 11:36:27,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:36:27,804 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 11:36:27,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:36:28,065 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 11:36:28,066 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 11:36:28,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 11:36:28.358 UTC [11276] WARNING: there is no transaction in progress
2026-09-26 11:36:28,365 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 11:36:28,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:36:28,673 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 11:36:28,673 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 11:36:28,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 11:36:28,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 11:36:29,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4pw5.c.gcp-upstream.internal','N') ;"
2026-09-26 11:36:29,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 11:36:29,734 [I] Table x_portal_user does not exist in database ranger
2026-09-26 11:36:29,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 11:36:30,004 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 11:36:30,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 11:36:30,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 11:36:30,566 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 11:36:30,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 11:36:31.737 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.758 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.779 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.800 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.819 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.842 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.917 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.926 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.943 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.956 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.969 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:31.987 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:32.001 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:32.010 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:32.020 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:32.029 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 11:36:32,895 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 11:36:32,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 11:36:33,175 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 11:36:33,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 11:36:33,441 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 11:36:33,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 11:36:33,725 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 11:36:33,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 11:36:34,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 11:36:34,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4pw5.c.gcp-upstream.internal';"
2026-09-26 11:36:34,510 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 11:36:34,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 11:36:34,763 [I] DB_PATCHES have already been applied
I20260926 11:36:34.769723 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPt_vxu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 11:36:34.769804 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 11:36:34.773656 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:42157
Sep 26, 2026 11:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 11:36:35.948457 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.732s user 0.008s sys 0.004s
Sep 26, 2026 11:36:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:36:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 11:36:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:36:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:36:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 11:36:58.279985 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 11:36:58 dist-test-slave-4pw5 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422618, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 11:36:58.376868 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 11:36:58.387051 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 11:36:59.441278 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 11:36:59.441397 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-kms
I20260926 11:36:59.441514 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:57501
2026-09-26 11:36:59,651 [I] DB FLAVOR :POSTGRES
2026-09-26 11:36:59,651 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 11:36:59,652 [I] Checking connection
2026-09-26 11:36:59,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 11:36:59,932 [I] connection success
2026-09-26 11:36:59,932 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 11:36:59,932 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 11:36:59,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 11:37:00,186 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 11:37:00,187 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 11:37:00,187 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 11:37:00,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 11:37:00,485 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 11:37:01.044513 392 mini_ranger_kms.cc:310] Created kms service
I20260926 11:37:01.272143 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 11:37:01.333428 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 11:37:01 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422618, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:37:01.856129 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 11:37:01.856305 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 11:37:01.856335 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 11:37:01.861379 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11565
I20260926 11:37:01.861560 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39669
Sep 26, 2026 11:37:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:37:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:37:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:37:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:37:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:37:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 11:37:04.234611 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.858s user 0.002s sys 0.193s
I20260926 11:37:04.234721 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 11:37:04.234855 392 mini_ranger_kms.cc:401] 127.0.98.20:39669/kms/v1/keys
Sep 26 11:37:04 dist-test-slave-4pw5 krb5kdc[11171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422624, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 11:37:04 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422624, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 11:37:05 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422624, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 11:37:07 dist-test-slave-4pw5 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:37:07.789789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46409
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46409
--ranger_config_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39669/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:37:07.922516 11615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:37:07.922819 11615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:37:07.922860 11615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:37:07.927161 11615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:37:07.927246 11615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:37:07.927266 11615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:37:07.927284 11615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:37:07.927340 11615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:37:07.933324 11615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39669/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46409
--ranger_config_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:37:07.934520 11615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:37:07.935640 11615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:37:07.946739 11621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:37:07.946729 11620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:37:07.946919 11615 server_base.cc:1034] running on GCE node
W20260926 11:37:07.946729 11623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:37:07.947901 11615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:37:07.949174 11615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:37:07.950412 11615 hybrid_clock.cc:648] HybridClock initialized: now 1790422627950373 us; error 55 us; skew 500 ppm
Sep 26 11:37:07 dist-test-slave-4pw5 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:37:07.958567 11615 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:37:07.960060 11615 webserver.cc:460] Webserver started at http://127.0.98.62:39397/ using document root <none> and password file <none>
I20260926 11:37:07.960659 11615 fs_manager.cc:359] Metadata directory not provided
I20260926 11:37:07.960745 11615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:37:07.960991 11615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:37:07 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:37:08.027640 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "7998f5aa1e6048b7b19aca11f4a35a96"
format_stamp: "Formatted at 2026-09-26 11:37:08 on dist-test-slave-4pw5"
server_key: "edfa9d52806d5cd019c297621af134a9"
server_key_iv: "49b04ffd43f59567ce049f0178191c06"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:08.028435 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "7998f5aa1e6048b7b19aca11f4a35a96"
format_stamp: "Formatted at 2026-09-26 11:37:08 on dist-test-slave-4pw5"
server_key: "edfa9d52806d5cd019c297621af134a9"
server_key_iv: "49b04ffd43f59567ce049f0178191c06"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:08.033258 11615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:37:08.051699 11630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:08.054030 11615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:37:08.054221 11615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "7998f5aa1e6048b7b19aca11f4a35a96"
format_stamp: "Formatted at 2026-09-26 11:37:08 on dist-test-slave-4pw5"
server_key: "edfa9d52806d5cd019c297621af134a9"
server_key_iv: "49b04ffd43f59567ce049f0178191c06"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:08.054394 11615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:37:08.072351 11615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:37:08.075562 11615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:37:08.075852 11615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:37:08.084831 11615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46409
I20260926 11:37:08.084863 11682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46409 every 8 connection(s)
I20260926 11:37:08.085916 11615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:37:08.088907 11683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:37:08.090435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 11615
I20260926 11:37:08.090562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:37:08.090992 392 external_mini_cluster.cc:1428] Setting key c7d0b778aa4776fa33e8bd4830db1e83
I20260926 11:37:08.096228 11683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96: Bootstrap starting.
I20260926 11:37:08.098938 11683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96: Neither blocks nor log segments found. Creating new log.
I20260926 11:37:08.099970 11683 log.cc:824] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96: Log is configured to *not* fsync() on all Append() calls
I20260926 11:37:08.102305 11683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96: No bootstrap required, opened a new log
Sep 26 11:37:08 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:37:08.107066 11683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998f5aa1e6048b7b19aca11f4a35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46409 } }
I20260926 11:37:08.107385 11683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:37:08.107437 11683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7998f5aa1e6048b7b19aca11f4a35a96, State: Initialized, Role: FOLLOWER
I20260926 11:37:08.108032 11683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998f5aa1e6048b7b19aca11f4a35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46409 } }
I20260926 11:37:08.108351 11683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:37:08.108443 11683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:37:08.108552 11683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:37:08.109771 11683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998f5aa1e6048b7b19aca11f4a35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46409 } }
I20260926 11:37:08.109956 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:08.092779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.0.98.62:46409)
0926 11:37:08.093260 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:08.093279 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:08.093323 (+ 44us) server_negotiation.cc:407] Connection header received
0926 11:37:08.094250 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:08.094278 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:08.094732 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:08.095204 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:37:08.096290 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:08.097771 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:08.099058 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:08.099784 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:08.099963 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:08.103090 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:37:08.103097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:37:08.103115 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:37:08.103146 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:37:08.105923 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:08.106564 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:08.106570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:08.106577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:08.106708 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:08.107023 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:08.107026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:08.107028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:08.107609 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:37:08.108459 (+ 850us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:08.108626 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 11:37:08.108958 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":142,"threads_started":1}
I20260926 11:37:08.110157 11683 leader_election.cc:302] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7998f5aa1e6048b7b19aca11f4a35a96; no voters:
I20260926 11:37:08.110565 11683 leader_election.cc:288] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:37:08.110677 11688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:37:08.110893 11688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [term 1 LEADER]: Becoming Leader. State: Replica: 7998f5aa1e6048b7b19aca11f4a35a96, State: Running, Role: LEADER
I20260926 11:37:08.111238 11688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998f5aa1e6048b7b19aca11f4a35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46409 } }
I20260926 11:37:08.112131 11683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:37:08.112576 11688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7998f5aa1e6048b7b19aca11f4a35a96. Latest consensus state: current_term: 1 leader_uuid: "7998f5aa1e6048b7b19aca11f4a35a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998f5aa1e6048b7b19aca11f4a35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46409 } } }
I20260926 11:37:08.112725 11688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [sys.catalog]: This master's current role is: LEADER
I20260926 11:37:08.113308 11689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7998f5aa1e6048b7b19aca11f4a35a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998f5aa1e6048b7b19aca11f4a35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46409 } } }
I20260926 11:37:08.113413 11689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96 [sys.catalog]: This master's current role is: LEADER
I20260926 11:37:08.113662 11692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:37:08.117157 11692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:37:08.117586 11683 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 11:37:08.135950 11692 catalog_manager.cc:1347] Generated new cluster ID: 2a760117fc1246ce9d2fd69506abbd9a
I20260926 11:37:08.136065 11692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:37:08.149066 11692 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:37:08.150624 11692 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:37:08.161839 11692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96: Generated new TSK 0
I20260926 11:37:08.162920 11692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:37:08.976020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39669/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46409
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:37:09.137815 11728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:37:09.138242 11728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:37:09.138394 11728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:37:09.145810 11728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:37:09.145924 11728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:37:09.146029 11728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:37:09.155556 11728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39669/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:37:09.157120 11728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:37:09.158609 11728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:37:09.171237 11735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:37:09.171648 11728 server_base.cc:1034] running on GCE node
W20260926 11:37:09.171324 11738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:37:09.171969 11736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:37:09.172523 11728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:37:09.173430 11728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:37:09.174638 11728 hybrid_clock.cc:648] HybridClock initialized: now 1790422629174614 us; error 42 us; skew 500 ppm
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:37:09.179044 11728 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:37:09.180357 11728 webserver.cc:460] Webserver started at http://127.0.98.1:46799/ using document root <none> and password file <none>
I20260926 11:37:09.181139 11728 fs_manager.cc:359] Metadata directory not provided
I20260926 11:37:09.181234 11728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:37:09.181483 11728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:37:09.199213 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "7fd393d3cdea412498c55f004e72bfa9"
format_stamp: "Formatted at 2026-09-26 11:37:09 on dist-test-slave-4pw5"
server_key: "83b43ac2ad4338f3c275718c75d60cd8"
server_key_iv: "09ff59c4388c9f1d13c23383ca6c9641"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:09.200133 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7fd393d3cdea412498c55f004e72bfa9"
format_stamp: "Formatted at 2026-09-26 11:37:09 on dist-test-slave-4pw5"
server_key: "83b43ac2ad4338f3c275718c75d60cd8"
server_key_iv: "09ff59c4388c9f1d13c23383ca6c9641"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:09.209316 11728 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260926 11:37:09.236084 11745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:09.238070 11728 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260926 11:37:09.238261 11728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "7fd393d3cdea412498c55f004e72bfa9"
format_stamp: "Formatted at 2026-09-26 11:37:09 on dist-test-slave-4pw5"
server_key: "83b43ac2ad4338f3c275718c75d60cd8"
server_key_iv: "09ff59c4388c9f1d13c23383ca6c9641"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:09.238440 11728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:37:09.269840 11728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:37:09.274462 11728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:37:09.274742 11728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:37:09.275549 11728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:37:09.276750 11728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:37:09.276859 11728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:09.277000 11728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:37:09.277081 11728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:09.288462 11728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38227
I20260926 11:37:09.289811 11728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:37:09.296605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 11728
I20260926 11:37:09.296768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:37:09.297137 392 external_mini_cluster.cc:1428] Setting key a99e10e8876912d9e85f5ba65ffc26f2
I20260926 11:37:09.298193 11860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38227 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:37:09.322257 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:09.301287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52709 (local address 127.0.98.62:46409)
0926 11:37:09.303075 (+ 1788us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:09.303080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:09.303098 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:37:09.303146 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:09.303149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:09.303218 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:09.303342 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:37:09.305243 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:09.306214 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:09.308236 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:09.308654 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:09.308768 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:09.316131 (+ 7363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:37:09.316137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:37:09.316142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:37:09.316183 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:37:09.318369 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:09.319466 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:09.319471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:09.319488 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:09.319657 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:09.320412 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:09.320416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:09.320420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:09.320752 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:37:09.320868 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:09.321803 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 11:37:09.321996 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1541,"thread_start_us":104,"threads_started":1}
I20260926 11:37:09.324651 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:09.300000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46409 (local address 127.0.98.1:52709)
0926 11:37:09.301379 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:09.301447 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 11:37:09.302714 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:37:09.303761 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:37:09.303773 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:37:09.304181 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:37:09.305017 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:09.305045 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:09.306414 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:09.306420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:09.308053 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:09.308067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:09.309002 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:09.309010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:09.309269 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:09.310069 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:37:09.310099 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:37:09.315802 (+ 5703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:37:09.318566 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:37:09.318577 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:37:09.318604 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:37:09.319293 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:37:09.319823 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:37:09.319830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:37:09.319832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:37:09.320256 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:37:09.320968 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:37:09.320977 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:37:09.321454 (+ 477us) client_negotiation.cc:769] Sending connection context
0926 11:37:09.322955 (+ 1501us) client_negotiation.cc:240] Negotiation successful
0926 11:37:09.323423 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"thread_start_us":160,"threads_started":1}
I20260926 11:37:09.326133 11861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46409
I20260926 11:37:09.326439 11861 heartbeater.cc:464] Registering TS with master...
I20260926 11:37:09.327064 11861 heartbeater.cc:511] Master 127.0.98.62:46409 requested a full tablet report, sending...
I20260926 11:37:09.328845 11647 ts_manager.cc:194] Registered new tserver with Master: 7fd393d3cdea412498c55f004e72bfa9 (127.0.98.1:38227)
I20260926 11:37:09.330127 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:37:09.335820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39669/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46409
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 11:37:09.498306 11867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:37:09.498795 11867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:37:09.498926 11867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:37:09.505481 11867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:37:09.505578 11867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:37:09.505668 11867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:37:09.513927 11867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39669/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:37:09.515391 11867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:37:09.516824 11867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:37:09.529930 11874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:37:09.530256 11876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:37:09.530040 11873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:37:09.530925 11867 server_base.cc:1034] running on GCE node
I20260926 11:37:09.531478 11867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:37:09.532433 11867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:37:09.533653 11867 hybrid_clock.cc:648] HybridClock initialized: now 1790422629533632 us; error 39 us; skew 500 ppm
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:37:09.538048 11867 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:37:09.539263 11867 webserver.cc:460] Webserver started at http://127.0.98.2:43093/ using document root <none> and password file <none>
I20260926 11:37:09.540071 11867 fs_manager.cc:359] Metadata directory not provided
I20260926 11:37:09.540161 11867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:37:09.540406 11867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:37:09.563017 11867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7"
format_stamp: "Formatted at 2026-09-26 11:37:09 on dist-test-slave-4pw5"
server_key: "3c90b25c55bcbfdbc73eb042c35868c4"
server_key_iv: "9b1978cbe5cbc6fb52c64938326d51fa"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:09.564018 11867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7"
format_stamp: "Formatted at 2026-09-26 11:37:09 on dist-test-slave-4pw5"
server_key: "3c90b25c55bcbfdbc73eb042c35868c4"
server_key_iv: "9b1978cbe5cbc6fb52c64938326d51fa"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:09.573359 11867 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.000s
I20260926 11:37:09.596638 11883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:09.600908 11867 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260926 11:37:09.601125 11867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7"
format_stamp: "Formatted at 2026-09-26 11:37:09 on dist-test-slave-4pw5"
server_key: "3c90b25c55bcbfdbc73eb042c35868c4"
server_key_iv: "9b1978cbe5cbc6fb52c64938326d51fa"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:09.601307 11867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:37:09.631018 11867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:37:09.636874 11867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:37:09.637161 11867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:37:09.638088 11867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:37:09.639673 11867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:37:09.639816 11867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:09.639943 11867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:37:09.639985 11867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:09.656888 11867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43929
I20260926 11:37:09.658370 11867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:37:09.664435 11996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43929 every 8 connection(s)
I20260926 11:37:09.667026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 11867
I20260926 11:37:09.667171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:37:09.667490 392 external_mini_cluster.cc:1428] Setting key 16ba98767f9695f1ed149a68e97242ee
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:37:09.708925 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:09.675765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42483 (local address 127.0.98.62:46409)
0926 11:37:09.675929 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:09.675934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:09.680243 (+ 4309us) server_negotiation.cc:407] Connection header received
0926 11:37:09.681685 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:09.681689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:09.681753 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:09.681807 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:37:09.686384 (+ 4577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:09.687245 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:09.695015 (+ 7770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:09.695445 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:09.695552 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:09.701425 (+ 5873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:37:09.701430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:37:09.701434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:37:09.701479 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:37:09.703542 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:09.706751 (+ 3209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:09.706756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:09.706758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:09.706869 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:09.707356 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:09.707360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:09.707362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:09.707696 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:37:09.707786 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:09.708586 (+ 800us) server_negotiation.cc:299] Negotiation successful
0926 11:37:09.708749 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:37:09.710685 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:09.674441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46409 (local address 127.0.98.2:42483)
0926 11:37:09.680004 (+ 5563us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:09.680097 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 11:37:09.681371 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:37:09.682286 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:37:09.682297 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:37:09.682692 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:37:09.684750 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:09.684774 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:09.687709 (+ 2935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:09.687712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:09.691118 (+ 3406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:09.691139 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:09.695892 (+ 4753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:09.695898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:09.696176 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:09.697818 (+ 1642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:37:09.697846 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:37:09.700995 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:37:09.706076 (+ 5081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:37:09.706084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:37:09.706107 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:37:09.706592 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:37:09.706998 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:37:09.707000 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:37:09.707003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:37:09.707230 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:37:09.707864 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:37:09.707871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:37:09.708265 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 11:37:09.709202 (+ 937us) client_negotiation.cc:240] Negotiation successful
0926 11:37:09.709593 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4722,"thread_start_us":185,"threads_started":1}
I20260926 11:37:09.712169 11997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46409
I20260926 11:37:09.712456 11997 heartbeater.cc:464] Registering TS with master...
I20260926 11:37:09.713083 11997 heartbeater.cc:511] Master 127.0.98.62:46409 requested a full tablet report, sending...
I20260926 11:37:09.714316 11647 ts_manager.cc:194] Registered new tserver with Master: 9ab5aaf0b4854e61802f8c1bdc6347a7 (127.0.98.2:43929)
I20260926 11:37:09.715103 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:37:09.739876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39669/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46409
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 11:37:09.918911 12002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:37:09.919303 12002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:37:09.919371 12002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:37:09.925804 12002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:37:09.925897 12002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:37:09.925990 12002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:37:09.934628 12002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39669/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:37:09.936203 12002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:37:09.937649 12002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:37:09.952292 12008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:37:09.952837 12002 server_base.cc:1034] running on GCE node
W20260926 11:37:09.952986 12007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:37:09.953306 12010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:37:09.954866 12002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:37:09.955798 12002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:37:09.971544 12002 hybrid_clock.cc:648] HybridClock initialized: now 1790422629971298 us; error 240 us; skew 500 ppm
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:37:09.976006 12002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:37:09.977357 12002 webserver.cc:460] Webserver started at http://127.0.98.3:34957/ using document root <none> and password file <none>
I20260926 11:37:09.978175 12002 fs_manager.cc:359] Metadata directory not provided
I20260926 11:37:09.978338 12002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:37:09.978747 12002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:37:09 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:37:10.008853 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "7c11aff08e5c4ea79ed3864367d4aa33"
format_stamp: "Formatted at 2026-09-26 11:37:10 on dist-test-slave-4pw5"
server_key: "0d7eea1f79bf506dd66bd5662e7306fa"
server_key_iv: "0be60cdfb9a79d0e8c3eebcf765e703d"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:10.012542 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7c11aff08e5c4ea79ed3864367d4aa33"
format_stamp: "Formatted at 2026-09-26 11:37:10 on dist-test-slave-4pw5"
server_key: "0d7eea1f79bf506dd66bd5662e7306fa"
server_key_iv: "0be60cdfb9a79d0e8c3eebcf765e703d"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:10.021113 12002 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.002s
I20260926 11:37:10.052920 12017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:10.057106 12002 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260926 11:37:10.057394 12002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "7c11aff08e5c4ea79ed3864367d4aa33"
format_stamp: "Formatted at 2026-09-26 11:37:10 on dist-test-slave-4pw5"
server_key: "0d7eea1f79bf506dd66bd5662e7306fa"
server_key_iv: "0be60cdfb9a79d0e8c3eebcf765e703d"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:10.057658 12002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:37:10.081550 12002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:37:10.085445 12002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:37:10.085635 12002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:37:10.086314 12002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:37:10.087363 12002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:37:10.087417 12002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:10.087468 12002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:37:10.087488 12002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:10.101287 12002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33419
I20260926 11:37:10.101330 12130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33419 every 8 connection(s)
I20260926 11:37:10.102504 12002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:37:10.106550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 12002
I20260926 11:37:10.106686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:37:10.107134 392 external_mini_cluster.cc:1428] Setting key 2754c03553957a47fc41ff4c04592cd0
Sep 26 11:37:10 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:37:10.124441 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.108117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55025 (local address 127.0.98.62:46409)
0926 11:37:10.108354 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:10.108359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:10.108376 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:37:10.108421 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:10.108424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:10.108476 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:10.108570 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:37:10.110341 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.111285 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.113895 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.114359 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.114472 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.119087 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:37:10.119093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:37:10.119097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:37:10.119148 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:37:10.121433 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:10.122172 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:10.122176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:10.122178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:10.122290 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:10.122768 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:10.122775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:10.122777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:10.123145 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:37:10.123267 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:10.124082 (+ 815us) server_negotiation.cc:299] Negotiation successful
0926 11:37:10.124250 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260926 11:37:10.126605 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.105313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46409 (local address 127.0.98.3:55025)
0926 11:37:10.105895 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:10.105962 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 11:37:10.107057 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:37:10.108922 (+ 1865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:37:10.108932 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:37:10.109361 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:37:10.110140 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.110163 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.111471 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.111474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.113668 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.113689 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.114774 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.114780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.115076 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.115993 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:37:10.116025 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:37:10.118763 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:37:10.121578 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:37:10.121587 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:37:10.121609 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:37:10.122028 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:37:10.122405 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:37:10.122408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:37:10.122410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:37:10.122635 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:37:10.123337 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:37:10.123345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:37:10.123773 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 11:37:10.125070 (+ 1297us) client_negotiation.cc:240] Negotiation successful
0926 11:37:10.125484 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":142,"threads_started":1}
I20260926 11:37:10.128471 12131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46409
I20260926 11:37:10.128742 12131 heartbeater.cc:464] Registering TS with master...
I20260926 11:37:10.129426 12131 heartbeater.cc:511] Master 127.0.98.62:46409 requested a full tablet report, sending...
I20260926 11:37:10.130681 11647 ts_manager.cc:194] Registered new tserver with Master: 7c11aff08e5c4ea79ed3864367d4aa33 (127.0.98.3:33419)
I20260926 11:37:10.131398 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55025
I20260926 11:37:10.135624 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:37:10.152364 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.140084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.0.98.62:46409)
0926 11:37:10.140207 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:10.140212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:10.140773 (+ 561us) server_negotiation.cc:407] Connection header received
0926 11:37:10.140981 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:10.140984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:10.141035 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:10.141088 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:37:10.142092 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.142886 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.144275 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.144605 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.144686 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.147480 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:37:10.147485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:37:10.147488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:37:10.147525 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:37:10.149519 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:10.150102 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:10.150106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:10.150107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:10.150215 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:10.150543 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:10.150547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:10.150548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:10.150781 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:37:10.150889 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:10.152052 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0926 11:37:10.152198 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:37:10.156677 11647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:37:10.161587 11647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:37:10.193548 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.176221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38227 (local address 127.0.0.1:39554)
0926 11:37:10.182352 (+ 6131us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:10.182437 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 11:37:10.185126 (+ 2689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:37:10.188466 (+ 3340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:37:10.188470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:37:10.188488 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:37:10.188580 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.188586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.190017 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.190021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.191519 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.191530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.192941 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.192945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.193066 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.193109 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 11:37:10.193233 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 11:37:10.193379 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5883,"thread_start_us":119,"threads_started":1}
I20260926 11:37:10.194536 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.176332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.0.98.1:38227)
0926 11:37:10.187799 (+ 11467us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:10.187807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:10.187835 (+ 28us) server_negotiation.cc:407] Connection header received
0926 11:37:10.187913 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:10.187918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:10.188165 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:10.188311 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:37:10.188719 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.189867 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.191700 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.192579 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.192691 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.192858 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:10.194256 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0926 11:37:10.194391 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11347,"thread_start_us":65,"threads_started":1}
I20260926 11:37:10.197603 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.178677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43929 (local address 127.0.0.1:39004)
0926 11:37:10.183893 (+ 5216us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:10.183908 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:37:10.184044 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:37:10.185258 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:37:10.185265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:37:10.185291 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:37:10.185450 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.185457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.193130 (+ 7673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.193133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.194345 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.194357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.197142 (+ 2785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.197147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.197250 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.197277 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 11:37:10.197352 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 11:37:10.197451 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5147,"thread_start_us":121,"threads_started":1}
I20260926 11:37:10.198220 11779 tablet_service.cc:1467] Processing CreateTablet for tablet 249b6a4993494fb8b8d8167b1fac4153 (DEFAULT_TABLE table=test-table [id=b34e94aad9f54648be2ab7870ddbcdfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:37:10.199510 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249b6a4993494fb8b8d8167b1fac4153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:37:10.199348 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.181004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33419 (local address 127.0.0.1:54816)
0926 11:37:10.185683 (+ 4679us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:10.185697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:37:10.185772 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:37:10.187485 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:37:10.187488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:37:10.187506 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:37:10.187642 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.187649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.191008 (+ 3359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.191013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.192298 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.192309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.198931 (+ 6622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.198935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.199045 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.199070 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 11:37:10.199135 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 11:37:10.199212 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4614,"thread_start_us":110,"threads_started":1}
I20260926 11:37:10.200109 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.181451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54816 (local address 127.0.98.3:33419)
0926 11:37:10.183735 (+ 2284us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:10.183749 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:10.185872 (+ 2123us) server_negotiation.cc:407] Connection header received
0926 11:37:10.185961 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:10.185967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:10.187155 (+ 1188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:10.187320 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:37:10.187776 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.189025 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.194578 (+ 5553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.198574 (+ 3996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.198702 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.198860 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:10.199833 (+ 973us) server_negotiation.cc:299] Negotiation successful
0926 11:37:10.199958 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2155,"thread_start_us":99,"threads_started":1}
I20260926 11:37:10.203670 12064 tablet_service.cc:1467] Processing CreateTablet for tablet 249b6a4993494fb8b8d8167b1fac4153 (DEFAULT_TABLE table=test-table [id=b34e94aad9f54648be2ab7870ddbcdfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:37:10.204972 12064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249b6a4993494fb8b8d8167b1fac4153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:37:10.212044 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.179138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.0.98.2:43929)
0926 11:37:10.184349 (+ 5211us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:10.184366 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:10.184393 (+ 27us) server_negotiation.cc:407] Connection header received
0926 11:37:10.184512 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:10.184518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:10.184804 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:10.184977 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:37:10.189193 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.192978 (+ 3785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.195899 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.196797 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.196916 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.197074 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:10.211711 (+ 14637us) server_negotiation.cc:299] Negotiation successful
0926 11:37:10.211870 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5100,"thread_start_us":123,"threads_started":1}
I20260926 11:37:10.218084 11931 tablet_service.cc:1467] Processing CreateTablet for tablet 249b6a4993494fb8b8d8167b1fac4153 (DEFAULT_TABLE table=test-table [id=b34e94aad9f54648be2ab7870ddbcdfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:37:10.219533 11931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249b6a4993494fb8b8d8167b1fac4153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:37:10.222738 12150 tablet_bootstrap.cc:492] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9: Bootstrap starting.
I20260926 11:37:10.225477 12150 tablet_bootstrap.cc:654] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9: Neither blocks nor log segments found. Creating new log.
I20260926 11:37:10.226517 12150 log.cc:824] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9: Log is configured to *not* fsync() on all Append() calls
I20260926 11:37:10.235863 12150 tablet_bootstrap.cc:492] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9: No bootstrap required, opened a new log
I20260926 11:37:10.235772 12152 tablet_bootstrap.cc:492] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7: Bootstrap starting.
I20260926 11:37:10.236292 12150 ts_tablet_manager.cc:1397] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260926 11:37:10.238649 12152 tablet_bootstrap.cc:654] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7: Neither blocks nor log segments found. Creating new log.
I20260926 11:37:10.239181 12153 tablet_bootstrap.cc:492] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: Bootstrap starting.
I20260926 11:37:10.240780 12152 log.cc:824] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7: Log is configured to *not* fsync() on all Append() calls
I20260926 11:37:10.241602 12150 raft_consensus.cc:348] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } }
I20260926 11:37:10.242142 12150 raft_consensus.cc:374] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:37:10.242276 12150 raft_consensus.cc:729] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd393d3cdea412498c55f004e72bfa9, State: Initialized, Role: FOLLOWER
I20260926 11:37:10.243104 12150 consensus_queue.cc:260] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } }
I20260926 11:37:10.244453 12153 tablet_bootstrap.cc:654] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: Neither blocks nor log segments found. Creating new log.
I20260926 11:37:10.244921 12150 ts_tablet_manager.cc:1428] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260926 11:37:10.245434 11861 heartbeater.cc:503] Master 127.0.98.62:46409 was elected leader, sending a full tablet report...
I20260926 11:37:10.245505 12153 log.cc:824] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: Log is configured to *not* fsync() on all Append() calls
I20260926 11:37:10.249630 12152 tablet_bootstrap.cc:492] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7: No bootstrap required, opened a new log
I20260926 11:37:10.249998 12152 ts_tablet_manager.cc:1397] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20260926 11:37:10.254709 12153 tablet_bootstrap.cc:492] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: No bootstrap required, opened a new log
I20260926 11:37:10.255113 12153 ts_tablet_manager.cc:1397] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20260926 11:37:10.255826 12152 raft_consensus.cc:348] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } }
I20260926 11:37:10.256263 12152 raft_consensus.cc:374] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:37:10.256376 12152 raft_consensus.cc:729] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab5aaf0b4854e61802f8c1bdc6347a7, State: Initialized, Role: FOLLOWER
I20260926 11:37:10.256980 12152 consensus_queue.cc:260] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } }
I20260926 11:37:10.258267 12152 ts_tablet_manager.cc:1428] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 11:37:10.258733 11997 heartbeater.cc:503] Master 127.0.98.62:46409 was elected leader, sending a full tablet report...
I20260926 11:37:10.259964 12153 raft_consensus.cc:348] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } }
I20260926 11:37:10.261097 12153 raft_consensus.cc:374] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:37:10.261164 12153 raft_consensus.cc:729] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c11aff08e5c4ea79ed3864367d4aa33, State: Initialized, Role: FOLLOWER
I20260926 11:37:10.263072 12153 consensus_queue.cc:260] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } }
I20260926 11:37:10.263978 12131 heartbeater.cc:503] Master 127.0.98.62:46409 was elected leader, sending a full tablet report...
I20260926 11:37:10.266347 12153 ts_tablet_manager.cc:1428] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
W20260926 11:37:10.298336 11862 tablet.cc:2369] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 11:37:10 dist-test-slave-4pw5 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 11:37:10.358361 12132 tablet.cc:2369] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:37:10.362300 12158 raft_consensus.cc:482] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:37:10.362563 12158 raft_consensus.cc:504] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } }
I20260926 11:37:10.364045 12158 leader_election.cc:288] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd393d3cdea412498c55f004e72bfa9 (127.0.98.1:38227), 9ab5aaf0b4854e61802f8c1bdc6347a7 (127.0.98.2:43929)
I20260926 11:37:10.369563 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.364254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38227 (local address 127.0.98.3:49617)
0926 11:37:10.364575 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:10.364590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:37:10.364681 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:37:10.365385 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:37:10.365388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:37:10.365408 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:37:10.365526 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.365531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.366513 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.366516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.368038 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.368058 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.369132 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.369135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.369239 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.369263 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:37:10.369332 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 11:37:10.369412 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260926 11:37:10.370663 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.364840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49617 (local address 127.0.98.1:38227)
0926 11:37:10.364976 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:10.364981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:10.364994 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:37:10.365044 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:10.365047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:10.365092 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:10.365161 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:37:10.365649 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.366382 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.368232 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.368873 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.368994 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.369068 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:10.370463 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0926 11:37:10.370551 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:37:10.371326 11815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249b6a4993494fb8b8d8167b1fac4153" candidate_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd393d3cdea412498c55f004e72bfa9" is_pre_election: true
I20260926 11:37:10.372044 11815 raft_consensus.cc:2454] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c11aff08e5c4ea79ed3864367d4aa33 in term 0.
I20260926 11:37:10.372639 12018 leader_election.cc:302] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c11aff08e5c4ea79ed3864367d4aa33, 7fd393d3cdea412498c55f004e72bfa9; no voters:
I20260926 11:37:10.373031 12158 raft_consensus.cc:2781] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:37:10.373106 12158 raft_consensus.cc:482] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:37:10.373143 12158 raft_consensus.cc:3037] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:37:10.374452 12158 raft_consensus.cc:504] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } }
I20260926 11:37:10.374825 12158 leader_election.cc:288] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [CANDIDATE]: Term 1 election: Requested vote from peers 7fd393d3cdea412498c55f004e72bfa9 (127.0.98.1:38227), 9ab5aaf0b4854e61802f8c1bdc6347a7 (127.0.98.2:43929)
I20260926 11:37:10.375300 11815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249b6a4993494fb8b8d8167b1fac4153" candidate_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd393d3cdea412498c55f004e72bfa9"
I20260926 11:37:10.375423 11815 raft_consensus.cc:3037] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:37:10.376670 11815 raft_consensus.cc:2454] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c11aff08e5c4ea79ed3864367d4aa33 in term 1.
I20260926 11:37:10.377075 12018 leader_election.cc:302] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c11aff08e5c4ea79ed3864367d4aa33, 7fd393d3cdea412498c55f004e72bfa9; no voters:
I20260926 11:37:10.377295 12158 raft_consensus.cc:2781] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:37:10.377518 12158 raft_consensus.cc:686] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 1 LEADER]: Becoming Leader. State: Replica: 7c11aff08e5c4ea79ed3864367d4aa33, State: Running, Role: LEADER
I20260926 11:37:10.377878 12158 consensus_queue.cc:237] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } }
I20260926 11:37:10.382510 11647 catalog_manager.cc:5495] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c11aff08e5c4ea79ed3864367d4aa33 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33419 } health_report { overall_health: HEALTHY } } }
I20260926 11:37:10.385270 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.364562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43929 (local address 127.0.98.3:43297)
0926 11:37:10.369750 (+ 5188us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:10.369765 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:37:10.369853 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:37:10.371033 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:37:10.371036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:37:10.371058 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:37:10.371139 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.371144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.372259 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.372262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.380731 (+ 8469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:37:10.380747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.384820 (+ 4073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.384824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:37:10.384922 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.384954 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 11:37:10.385020 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 11:37:10.385106 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5127,"thread_start_us":112,"threads_started":1}
I20260926 11:37:10.385756 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.365003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43297 (local address 127.0.98.2:43929)
0926 11:37:10.365123 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:10.365127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:10.369950 (+ 4823us) server_negotiation.cc:407] Connection header received
0926 11:37:10.370010 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:10.370012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:10.370058 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:10.370120 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:37:10.371272 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.372126 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.383845 (+ 11719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.384550 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.384643 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.384711 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:10.385548 (+ 837us) server_negotiation.cc:299] Negotiation successful
0926 11:37:10.385640 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:37:10.387287 11951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249b6a4993494fb8b8d8167b1fac4153" candidate_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7"
I20260926 11:37:10.387337 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249b6a4993494fb8b8d8167b1fac4153" candidate_uuid: "7c11aff08e5c4ea79ed3864367d4aa33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" is_pre_election: true
I20260926 11:37:10.387528 11951 raft_consensus.cc:3037] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:37:10.389062 11951 raft_consensus.cc:2454] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c11aff08e5c4ea79ed3864367d4aa33 in term 1.
W20260926 11:37:10.417304 11998 tablet.cc:2369] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:37:10.432566 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.425742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.98.3:33419)
0926 11:37:10.425873 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:10.425877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:10.425894 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:37:10.425998 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:10.426001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:10.426064 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:10.426176 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:37:10.426564 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.427380 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.428728 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:10.429090 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:10.429208 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:10.429727 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:37:10.430344 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:37:10.430470 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:10.432291 (+ 1821us) server_negotiation.cc:299] Negotiation successful
0926 11:37:10.432380 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:37:10.461316 11951 raft_consensus.cc:1261] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7 [term 1 FOLLOWER]: Refusing update from remote peer 7c11aff08e5c4ea79ed3864367d4aa33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:37:10.461397 11815 raft_consensus.cc:1261] T 249b6a4993494fb8b8d8167b1fac4153 P 7fd393d3cdea412498c55f004e72bfa9 [term 1 FOLLOWER]: Refusing update from remote peer 7c11aff08e5c4ea79ed3864367d4aa33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:37:10.462258 12161 consensus_queue.cc:1035] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd393d3cdea412498c55f004e72bfa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:37:10.462801 12161 consensus_queue.cc:1035] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ab5aaf0b4854e61802f8c1bdc6347a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:37:10.499311 12166 mvcc.cc:205] Tried to move back new op lower bound from 7333571094306623488 to 7333571094030274560. Current Snapshot: MvccSnapshot[applied={T|T < 7333571094306623488}]
I20260926 11:37:10.499624 12167 mvcc.cc:205] Tried to move back new op lower bound from 7333571094306623488 to 7333571094030274560. Current Snapshot: MvccSnapshot[applied={T|T < 7333571094306623488}]
I20260926 11:37:10.513159 11646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:37:10.513374 11646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:37:10.516443 11646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7998f5aa1e6048b7b19aca11f4a35a96: Sending DeleteTablet for 3 replicas of tablet 249b6a4993494fb8b8d8167b1fac4153
I20260926 11:37:10.517659 11779 tablet_service.cc:1514] Processing DeleteTablet for tablet 249b6a4993494fb8b8d8167b1fac4153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:37:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39554
I20260926 11:37:10.518429 11931 tablet_service.cc:1514] Processing DeleteTablet for tablet 249b6a4993494fb8b8d8167b1fac4153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:37:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39004
I20260926 11:37:10.519618 12064 tablet_service.cc:1514] Processing DeleteTablet for tablet 249b6a4993494fb8b8d8167b1fac4153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:37:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54816
I20260926 11:37:10.521025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 11728
I20260926 11:37:10.522151 12175 tablet_replica.cc:330] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7: stopping tablet replica
I20260926 11:37:10.522506 12175 raft_consensus.cc:2229] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:37:10.522854 12175 raft_consensus.cc:2258] T 249b6a4993494fb8b8d8167b1fac4153 P 9ab5aaf0b4854e61802f8c1bdc6347a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:37:10.523339 12176 tablet_replica.cc:330] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: stopping tablet replica
I20260926 11:37:10.523756 12176 raft_consensus.cc:2229] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:37:10.524219 12176 raft_consensus.cc:2258] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 11:37:10.529783 11631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38227 (error 108)
I20260926 11:37:10.530690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 11867
I20260926 11:37:10.534886 12176 ts_tablet_manager.cc:1905] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:37:10.543422 12176 ts_tablet_manager.cc:1918] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:37:10.543699 12176 log.cc:1196] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/249b6a4993494fb8b8d8167b1fac4153
I20260926 11:37:10.544178 12176 ts_tablet_manager.cc:1939] T 249b6a4993494fb8b8d8167b1fac4153 P 7c11aff08e5c4ea79ed3864367d4aa33: Deleting consensus metadata
I20260926 11:37:10.544780 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:10.544344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38227 (local address 127.0.0.1:39560)
0926 11:37:10.544603 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:10.544686 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
W20260926 11:37:10.545199 11631 catalog_manager.cc:4584] TS 7fd393d3cdea412498c55f004e72bfa9 (127.0.98.1:38227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 249b6a4993494fb8b8d8167b1fac4153: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38227: connect: Connection refused (error 111)
W20260926 11:37:10.549152 12144 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:37:10.545787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43929 (local address 127.0.0.1:39012)
0926 11:37:10.545901 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 11:37:10.545917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:37:10.546022 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:37:10.549027 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 11:37:10.549547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 12002
I20260926 11:37:10.557883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 11615
I20260926 11:37:10.566396 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 11:37:10.912436 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 11:37:10.912577 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 11:37:11.258416 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 11:37:11.258 UTC [11201] LOG: received smart shutdown request
2026-09-26 11:37:11.263 UTC [11201] LOG: background worker "logical replication launcher" (PID 11210) exited with exit code 1
2026-09-26 11:37:11.263 UTC [11205] LOG: shutting down
2026-09-26 11:37:11.502 UTC [11201] LOG: database system is shut down
2026-09-26T11:37:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50611 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:37:11 dist-test-slave-4pw5 krb5kdc[12186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:37:11 dist-test-slave-4pw5 krb5kdc[12186](info): set up 2 sockets
Sep 26 11:37:11 dist-test-slave-4pw5 krb5kdc[12186](info): commencing operation
krb5kdc: starting...
W20260926 11:37:12.740075 392 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:37:12 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:37:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:37:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 11:37:12.935465 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPt_vxu/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/postgres -l logfile start
2026-09-26 11:37:15.875 UTC [12216] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 11:37:15.875 UTC [12216] LOG: listening on IPv4 address "127.0.98.20", port 35961
2026-09-26 11:37:15.880 UTC [12216] LOG: listening on Unix socket "/tmp/.s.PGSQL.35961"
2026-09-26 11:37:15.905 UTC [12219] LOG: database system was shut down at 2026-09-26 11:37:14 UTC
2026-09-26 11:37:15.910 UTC [12216] LOG: database system is ready to accept connections
I20260926 11:37:16.892050 392 mini_postgres.cc:96] Postgres bound to 35961
2026-09-26 11:37:16.898 UTC [12227] FATAL: database "slave" does not exist
127.0.98.20:35961 - accepting connections
I20260926 11:37:16.899444 392 mini_ranger.cc:161] Starting Ranger...
I20260926 11:37:16.916487 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 11:37:17.543382 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 11:37:17.543474 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-admin
2026-09-26 11:37:17,846 [I] DB FLAVOR :POSTGRES
2026-09-26 11:37:17,847 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 11:37:17,847 [I] Checking connection..
2026-09-26 11:37:17,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 11:37:18,113 [I] Checking connection passed.
2026-09-26 11:37:18,114 [I] --------- Verifying version history table ---------
2026-09-26 11:37:18,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:37:18,383 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 11:37:18,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:37:18,652 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 11:37:18,652 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 11:37:18,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 11:37:18.943 UTC [12291] WARNING: there is no transaction in progress
2026-09-26 11:37:18,952 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 11:37:18,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:37:19,211 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 11:37:19,212 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 11:37:19,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 11:37:19,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 11:37:19,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4pw5.c.gcp-upstream.internal','N') ;"
2026-09-26 11:37:20,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 11:37:20,264 [I] Table x_portal_user does not exist in database ranger
2026-09-26 11:37:20,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 11:37:20,563 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 11:37:20,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 11:37:20,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 11:37:21,099 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 11:37:21,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 11:37:22.247 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.266 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.285 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.304 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.325 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.348 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.413 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.422 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.439 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.452 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.463 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.479 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.491 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.499 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.507 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:22.515 UTC [12399] WARNING: there is no transaction in progress
2026-09-26 11:37:23,375 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 11:37:23,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 11:37:23,631 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 11:37:23,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 11:37:23,897 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 11:37:23,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 11:37:24,164 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 11:37:24,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 11:37:24,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 11:37:24,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4pw5.c.gcp-upstream.internal';"
2026-09-26 11:37:24,958 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 11:37:24,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 11:37:25,243 [I] DB_PATCHES have already been applied
I20260926 11:37:25.249795 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPt_vxu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 11:37:25.249977 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 11:37:25.253770 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59981
Sep 26, 2026 11:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:37:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:37:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 11:37:26.413623 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.514s user 0.009s sys 0.002s
Sep 26, 2026 11:37:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:37:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 11:37:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:37:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:37:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 11:37:50.370322 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 11:37:50 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422670, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 11:37:50.464772 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 11:37:50.475857 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 11:37:51.546538 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 11:37:51.546653 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-kms
I20260926 11:37:51.546677 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:35961
2026-09-26 11:37:51,774 [I] DB FLAVOR :POSTGRES
2026-09-26 11:37:51,775 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 11:37:51,775 [I] Checking connection
2026-09-26 11:37:51,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 11:37:52,084 [I] connection success
2026-09-26 11:37:52,085 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 11:37:52,085 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 11:37:52,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 11:37:52,359 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 11:37:52,360 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 11:37:52,360 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 11:37:52,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35961/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 11:37:52,654 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 11:37:53.197590 392 mini_ranger_kms.cc:310] Created kms service
I20260926 11:37:53.427186 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 11:37:53.479738 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 11:37:53 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422670, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:37:53.911968 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 11:37:53.912142 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPt_vxu/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPt_vxu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 11:37:53.912185 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 11:37:53.916647 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12580
I20260926 11:37:53.916781 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39413
Sep 26, 2026 11:37:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:37:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:37:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:37:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:37:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:37:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 11:37:56 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422676, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:37:56.226825 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.762s user 0.021s sys 0.179s
I20260926 11:37:56.226994 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 11:37:56.227063 392 mini_ranger_kms.cc:401] 127.0.98.20:39413/kms/v1/keys
Sep 26 11:37:56 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422676, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 11:37:57 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422676, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 11:37:59 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:37:59.415494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38799
--ranger_config_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39413/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:37:59.533012 12630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:37:59.533340 12630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:37:59.533382 12630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:37:59.537669 12630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:37:59.537761 12630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:37:59.537782 12630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:37:59.537798 12630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:37:59.537814 12630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:37:59.544739 12630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38799
--ranger_config_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:37:59.546078 12630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:37:59.547297 12630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:37:59.557544 12630 server_base.cc:1034] running on GCE node
W20260926 11:37:59.557565 12636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:37:59.557559 12635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:37:59.557525 12638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:37:59.558353 12630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:37:59.559652 12630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:37:59.560839 12630 hybrid_clock.cc:648] HybridClock initialized: now 1790422679560813 us; error 40 us; skew 500 ppm
Sep 26 11:37:59 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:37:59.569046 12630 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:37:59.570235 12630 webserver.cc:460] Webserver started at http://127.0.98.62:33743/ using document root <none> and password file <none>
I20260926 11:37:59.570837 12630 fs_manager.cc:359] Metadata directory not provided
I20260926 11:37:59.570897 12630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:37:59.571135 12630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:37:59 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:37:59.622108 12630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "c7b64f15a5a7472680728ec2037e0d6e"
format_stamp: "Formatted at 2026-09-26 11:37:59 on dist-test-slave-4pw5"
server_key: "545802ba46fc09c7bb38619fd214c0ed"
server_key_iv: "d86cfcc0596eeff9726e459dd5f55552"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:59.622792 12630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7b64f15a5a7472680728ec2037e0d6e"
format_stamp: "Formatted at 2026-09-26 11:37:59 on dist-test-slave-4pw5"
server_key: "545802ba46fc09c7bb38619fd214c0ed"
server_key_iv: "d86cfcc0596eeff9726e459dd5f55552"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:59.627041 12630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:37:59.643147 12645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:37:59.644691 12630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:37:59.644872 12630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "c7b64f15a5a7472680728ec2037e0d6e"
format_stamp: "Formatted at 2026-09-26 11:37:59 on dist-test-slave-4pw5"
server_key: "545802ba46fc09c7bb38619fd214c0ed"
server_key_iv: "d86cfcc0596eeff9726e459dd5f55552"
server_key_version: "kuduclusterkey@0"
I20260926 11:37:59.645052 12630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:37:59.660176 12630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:37:59.663661 12630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:37:59.663888 12630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:37:59.672904 12630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38799
I20260926 11:37:59.672911 12697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38799 every 8 connection(s)
I20260926 11:37:59.674023 12630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:37:59.674557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 12630
I20260926 11:37:59.674695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:37:59.675046 392 external_mini_cluster.cc:1428] Setting key 7e7228906cd623ed91124bb5f83eeac7
I20260926 11:37:59.677485 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:37:59.684975 12698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Bootstrap starting.
Sep 26 11:37:59 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:37:59.687649 12698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Neither blocks nor log segments found. Creating new log.
I20260926 11:37:59.688818 12698 log.cc:824] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Log is configured to *not* fsync() on all Append() calls
I20260926 11:37:59.690930 12698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: No bootstrap required, opened a new log
I20260926 11:37:59.691696 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:37:59.676583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.0.98.62:38799)
0926 11:37:59.677025 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 11:37:59.677044 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:37:59.677091 (+ 47us) server_negotiation.cc:407] Connection header received
0926 11:37:59.677829 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:37:59.677848 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:37:59.678204 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:37:59.678505 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:37:59.679514 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:59.680748 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:59.682001 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:37:59.682581 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:37:59.682730 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:37:59.686005 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:37:59.686011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:37:59.686026 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:37:59.686059 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:37:59.688099 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:59.688885 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:59.688895 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:59.688905 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:59.689059 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:37:59.689429 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:37:59.689434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:37:59.689436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:37:59.689907 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:37:59.690139 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 11:37:59.690468 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 11:37:59.690795 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":162,"threads_started":1}
I20260926 11:37:59.694588 12698 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260926 11:37:59.694826 12698 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:37:59.694896 12698 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b64f15a5a7472680728ec2037e0d6e, State: Initialized, Role: FOLLOWER
I20260926 11:37:59.695323 12698 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260926 11:37:59.695442 12698 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:37:59.695525 12698 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:37:59.695655 12698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:37:59.696671 12698 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260926 11:37:59.697043 12698 leader_election.cc:302] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7b64f15a5a7472680728ec2037e0d6e; no voters:
I20260926 11:37:59.697378 12698 leader_election.cc:288] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:37:59.697593 12703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:37:59.697870 12703 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 1 LEADER]: Becoming Leader. State: Replica: c7b64f15a5a7472680728ec2037e0d6e, State: Running, Role: LEADER
I20260926 11:37:59.698276 12703 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260926 11:37:59.698760 12698 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:37:59.699932 12704 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7b64f15a5a7472680728ec2037e0d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260926 11:37:59.700057 12704 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: This master's current role is: LEADER
I20260926 11:37:59.700186 12705 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7b64f15a5a7472680728ec2037e0d6e. Latest consensus state: current_term: 1 leader_uuid: "c7b64f15a5a7472680728ec2037e0d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260926 11:37:59.700268 12705 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: This master's current role is: LEADER
I20260926 11:37:59.700675 12712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:37:59.704042 12712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:37:59.704739 12698 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 11:37:59.715356 12712 catalog_manager.cc:1347] Generated new cluster ID: 8f9ceb818d0e4d4b89dcfb15641b861b
I20260926 11:37:59.715453 12712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:37:59.728129 12712 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:37:59.729655 12712 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:37:59.736397 12712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Generated new TSK 0
I20260926 11:37:59.737418 12712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:38:00.483392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:00.612681 12743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:00.613004 12743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:00.613121 12743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:00.619637 12743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:00.619781 12743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:00.619899 12743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:38:00.628736 12743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:00.630451 12743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:00.632026 12743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:00.646168 12750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:00.646862 12749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:00.648109 12743 server_base.cc:1034] running on GCE node
W20260926 11:38:00.653839 12752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:00.654667 12743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:00.655756 12743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:00.659117 12743 hybrid_clock.cc:648] HybridClock initialized: now 1790422680659001 us; error 122 us; skew 500 ppm
Sep 26 11:38:00 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:00.663934 12743 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:38:00.665470 12743 webserver.cc:460] Webserver started at http://127.0.98.1:44937/ using document root <none> and password file <none>
I20260926 11:38:00.666392 12743 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:00.666571 12743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:00.666909 12743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:38:00 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:38:00.689182 12743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "4a96d87aa0e740f493098a0becf988a3"
format_stamp: "Formatted at 2026-09-26 11:38:00 on dist-test-slave-4pw5"
server_key: "15f5b5c675839b8dddb36f7920cbb371"
server_key_iv: "51a3acfb22c7ab94d55676fcd4742b16"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:00.691869 12743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4a96d87aa0e740f493098a0becf988a3"
format_stamp: "Formatted at 2026-09-26 11:38:00 on dist-test-slave-4pw5"
server_key: "15f5b5c675839b8dddb36f7920cbb371"
server_key_iv: "51a3acfb22c7ab94d55676fcd4742b16"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:00.697721 12743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 11:38:00.723641 12760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.001s sys 0.000s
I20260926 11:38:00.726562 12743 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260926 11:38:00.726773 12743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "4a96d87aa0e740f493098a0becf988a3"
format_stamp: "Formatted at 2026-09-26 11:38:00 on dist-test-slave-4pw5"
server_key: "15f5b5c675839b8dddb36f7920cbb371"
server_key_iv: "51a3acfb22c7ab94d55676fcd4742b16"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:00.726953 12743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:00.765668 12743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:00.769796 12743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:00.770118 12743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:00.770961 12743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:00.772257 12743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:00.772363 12743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:00.772449 12743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:00.772480 12743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:00.784312 12743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34879
I20260926 11:38:00.784775 12875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34879 every 8 connection(s)
I20260926 11:38:00.785846 12743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:38:00.787714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 12743
I20260926 11:38:00.787838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:38:00.788121 392 external_mini_cluster.cc:1428] Setting key 3fdf9fec5fa9b1a7f79945530ae1995b
Sep 26 11:38:00 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:38:00.805218 12879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:00.789826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60537 (local address 127.0.98.62:38799)
0926 11:38:00.790067 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:00.790073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:00.790090 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:38:00.790686 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:00.790691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:00.790749 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:00.790862 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:00.792430 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:00.793362 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:00.795170 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:00.795529 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:00.795718 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:00.799881 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:00.799889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:00.799892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:00.799929 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:00.802095 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:00.802902 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:00.802906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:00.802908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:00.803029 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:00.803597 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:00.803601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:00.803603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:00.803885 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:00.803977 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:00.804855 (+ 878us) server_negotiation.cc:299] Negotiation successful
0926 11:38:00.805014 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260926 11:38:00.806545 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:00.788700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:60537)
0926 11:38:00.789307 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:00.789366 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:00.790418 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:00.791062 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:00.791073 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:00.791465 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:00.792250 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:00.792272 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:00.793533 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:00.793537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:00.794978 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:00.794990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:00.795773 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:00.795777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:00.795995 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:00.796780 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:00.796805 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:00.799544 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:00.802288 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:00.802297 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:00.802319 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:00.802755 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:00.803178 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:00.803182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:00.803183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:00.803431 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:00.804079 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:00.804088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:00.804501 (+ 413us) client_negotiation.cc:769] Sending connection context
0926 11:38:00.804784 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 11:38:00.805203 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":161,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:38:00.808259 12876 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260926 11:38:00.808533 12876 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:00.809175 12876 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260926 11:38:00.811183 12662 ts_manager.cc:194] Registered new tserver with Master: 4a96d87aa0e740f493098a0becf988a3 (127.0.98.1:34879)
I20260926 11:38:00.812853 12662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:38:00.830994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:38:00 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 11:38:00.977684 12882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:00.978032 12882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:00.978125 12882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:00.983032 12882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:00.983191 12882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:00.983335 12882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:38:00.990346 12882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:00.991798 12882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:00.992940 12882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:01.004830 12891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:01.004865 12889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:01.004876 12888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:01.005235 12882 server_base.cc:1034] running on GCE node
I20260926 11:38:01.005805 12882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:01.006757 12882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:01.007943 12882 hybrid_clock.cc:648] HybridClock initialized: now 1790422681007896 us; error 59 us; skew 500 ppm
Sep 26 11:38:01 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:01.011785 12882 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:38:01.013080 12882 webserver.cc:460] Webserver started at http://127.0.98.2:34097/ using document root <none> and password file <none>
I20260926 11:38:01.013888 12882 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:01.013998 12882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:01.014292 12882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:38:01 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:38:01.043176 12882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "5482f832f0304781bd83015d89084fc2"
format_stamp: "Formatted at 2026-09-26 11:38:01 on dist-test-slave-4pw5"
server_key: "a1258507c15e98d961c7fc03ba6d1dd8"
server_key_iv: "0ab2b8482f1b16f8effb80ef1e75a0c1"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:01.045621 12882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5482f832f0304781bd83015d89084fc2"
format_stamp: "Formatted at 2026-09-26 11:38:01 on dist-test-slave-4pw5"
server_key: "a1258507c15e98d961c7fc03ba6d1dd8"
server_key_iv: "0ab2b8482f1b16f8effb80ef1e75a0c1"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:01.052315 12882 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 11:38:01.076166 12898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:01.078047 12882 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260926 11:38:01.078238 12882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "5482f832f0304781bd83015d89084fc2"
format_stamp: "Formatted at 2026-09-26 11:38:01 on dist-test-slave-4pw5"
server_key: "a1258507c15e98d961c7fc03ba6d1dd8"
server_key_iv: "0ab2b8482f1b16f8effb80ef1e75a0c1"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:01.078410 12882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:01.098045 12882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:01.102344 12882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:01.102646 12882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:01.103538 12882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:01.104750 12882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:01.104862 12882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:01.104992 12882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:01.105057 12882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:01.134480 12882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43691
I20260926 11:38:01.136533 12882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:38:01.141359 13011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43691 every 8 connection(s)
I20260926 11:38:01.146025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 12882
I20260926 11:38:01.146152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:38:01.146459 392 external_mini_cluster.cc:1428] Setting key 8b0faf2deb74b2f34bedd629904737f2
Sep 26 11:38:01 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:38:01.167557 12879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:01.145277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40763 (local address 127.0.98.62:38799)
0926 11:38:01.145419 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:01.145424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:01.151028 (+ 5604us) server_negotiation.cc:407] Connection header received
0926 11:38:01.152744 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:01.152749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:01.152822 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:01.152885 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:01.154746 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:01.155748 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:01.157431 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:01.157870 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:01.158018 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:01.162153 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:01.162161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:01.162168 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:01.162215 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:01.164360 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:01.165233 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:01.165239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:01.165241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:01.165367 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:01.165917 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:01.165924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:01.165927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:01.166239 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:01.166337 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:01.167126 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 11:38:01.167330 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:38:01.168711 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:01.143993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:40763)
0926 11:38:01.150655 (+ 6662us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:01.150871 (+ 216us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:01.152393 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:01.153445 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:01.153457 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:01.153896 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:01.154526 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:01.154542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:01.156198 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:01.156204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:01.157247 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:01.157260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:01.158277 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:01.158285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:01.158659 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:01.159692 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:01.159722 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:01.161847 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:01.164555 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:01.164565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:01.164589 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:01.165076 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:01.165529 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:01.165533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:01.165535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:01.165779 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:01.166493 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:01.166501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:01.166843 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 11:38:01.167082 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 11:38:01.167470 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1674,"thread_start_us":184,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:38:01.170217 13012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260926 11:38:01.170487 13012 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:01.171149 13012 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260926 11:38:01.172573 12662 ts_manager.cc:194] Registered new tserver with Master: 5482f832f0304781bd83015d89084fc2 (127.0.98.2:43691)
I20260926 11:38:01.173560 12662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:38:01.208813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:38:01 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 11:38:01.383911 13017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:01.384349 13017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:01.384419 13017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:01.391029 13017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:01.391131 13017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:01.391222 13017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:38:01.399842 13017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:01.401353 13017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:01.402812 13017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:01.421684 13022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:01.422048 13025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:01.422566 13023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:01.422940 13017 server_base.cc:1034] running on GCE node
I20260926 11:38:01.423552 13017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:01.424402 13017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:01.425634 13017 hybrid_clock.cc:648] HybridClock initialized: now 1790422681425559 us; error 92 us; skew 500 ppm
Sep 26 11:38:01 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:01.429508 13017 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:38:01.430675 13017 webserver.cc:460] Webserver started at http://127.0.98.3:41185/ using document root <none> and password file <none>
I20260926 11:38:01.431394 13017 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:01.431471 13017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:01.431756 13017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:38:01 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:38:01.452337 13017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd"
format_stamp: "Formatted at 2026-09-26 11:38:01 on dist-test-slave-4pw5"
server_key: "c0a567e1b1c7f02392c45ef04248c0d2"
server_key_iv: "49bcc36f541e0585b4432cee270021b0"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:01.453157 13017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd"
format_stamp: "Formatted at 2026-09-26 11:38:01 on dist-test-slave-4pw5"
server_key: "c0a567e1b1c7f02392c45ef04248c0d2"
server_key_iv: "49bcc36f541e0585b4432cee270021b0"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:01.458698 13017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:38:01.477814 13032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:01.479418 13017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:38:01.479645 13017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd"
format_stamp: "Formatted at 2026-09-26 11:38:01 on dist-test-slave-4pw5"
server_key: "c0a567e1b1c7f02392c45ef04248c0d2"
server_key_iv: "49bcc36f541e0585b4432cee270021b0"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:01.479811 13017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:01.505064 13017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:01.509569 13017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:01.509852 13017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:01.510727 13017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:01.511933 13017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:01.512035 13017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:01.512130 13017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:01.512184 13017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:01.525727 13017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39875
I20260926 11:38:01.525787 13145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39875 every 8 connection(s)
I20260926 11:38:01.527123 13017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:38:01.528971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 13017
I20260926 11:38:01.529122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:38:01.529493 392 external_mini_cluster.cc:1428] Setting key ea8f4dcb9bedda09b8ee74da6862eaf8
Sep 26 11:38:01 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:01.547930 12879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:01.529526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54755 (local address 127.0.98.62:38799)
0926 11:38:01.529663 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:01.529668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:01.531146 (+ 1478us) server_negotiation.cc:407] Connection header received
0926 11:38:01.532545 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:01.532550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:01.532615 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:01.532711 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:01.534258 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:01.535122 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:01.536893 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:01.537254 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:01.537387 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:01.541049 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:01.541054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:01.541057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:01.541096 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:01.543723 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:01.544924 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:01.544928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:01.544932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:01.545059 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:01.546103 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:01.546108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:01.546110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:01.546429 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:01.546541 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:01.547518 (+ 977us) server_negotiation.cc:299] Negotiation successful
0926 11:38:01.547721 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:38:01.550263 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:01.530101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:54755)
0926 11:38:01.530960 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:01.531035 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:01.532274 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:01.532933 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:01.532942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:01.533320 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:01.534041 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:01.534060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:01.535280 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:01.535284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:01.536730 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:01.536744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:01.537434 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:01.537438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:01.537642 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:01.538442 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:01.538464 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:01.540788 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:01.544017 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:01.544036 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:01.544062 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:01.544574 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:01.545236 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:01.545241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:01.545243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:01.545517 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:01.546639 (+ 1122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:01.546649 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:01.547135 (+ 486us) client_negotiation.cc:769] Sending connection context
0926 11:38:01.548254 (+ 1119us) client_negotiation.cc:240] Negotiation successful
0926 11:38:01.548726 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":162,"threads_started":1}
I20260926 11:38:01.552199 13146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260926 11:38:01.552538 13146 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:01.553349 13146 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260926 11:38:01.554941 12662 ts_manager.cc:194] Registered new tserver with Master: 7b6b7f4deb1a44c98f8993fb18bd24bd (127.0.98.3:39875)
I20260926 11:38:01.555971 12662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54755
I20260926 11:38:01.557691 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:38:01.558692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 12743
I20260926 11:38:01.566476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 12882
I20260926 11:38:01.573518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 13017
I20260926 11:38:01.581305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 12630
I20260926 11:38:01.590263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--webserver_interface=127.0.98.62
--webserver_port=33743
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38799
--ranger_config_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39413/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39413/kms with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:01.717360 13149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:01.717737 13149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:01.717839 13149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:01.722177 13149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:01.722296 13149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:01.722338 13149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:01.722404 13149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:01.722447 13149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:01.728600 13149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38799
--ranger_config_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:01.729830 13149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:01.730976 13149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:01.740680 13155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:01.740700 13154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:01.741022 13149 server_base.cc:1034] running on GCE node
W20260926 11:38:01.740675 13157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:01.741791 13149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:01.743073 13149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:01.744310 13149 hybrid_clock.cc:648] HybridClock initialized: now 1790422681744273 us; error 61 us; skew 500 ppm
Sep 26 11:38:01 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:01.747741 13149 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:38:01.749023 13149 webserver.cc:460] Webserver started at http://127.0.98.62:33743/ using document root <none> and password file <none>
I20260926 11:38:01.749645 13149 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:01.749749 13149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 11:38:01 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:38:01.803725 13149 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:38:01.806411 13165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:01.808161 13149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:01.808367 13149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "c7b64f15a5a7472680728ec2037e0d6e"
format_stamp: "Formatted at 2026-09-26 11:37:59 on dist-test-slave-4pw5"
server_key: "545802ba46fc09c7bb38619fd214c0ed"
server_key_iv: "d86cfcc0596eeff9726e459dd5f55552"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:01.809125 13149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:01.832475 13149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:01.836440 13149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:01.836697 13149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:01.845510 13149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38799
I20260926 11:38:01.845520 13217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38799 every 8 connection(s)
I20260926 11:38:01.846779 13149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:38:01.849860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 13149
I20260926 11:38:01.850430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34879
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38799
--webserver_port=44937
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39413/kms with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:38:01.851642 13218 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:38:01.854029 13218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Bootstrap starting.
I20260926 11:38:01.873303 13218 log.cc:824] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:01.880769 13218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:38:01.881242 13218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Bootstrap complete.
I20260926 11:38:01.885989 13218 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260926 11:38:01.886485 13218 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b64f15a5a7472680728ec2037e0d6e, State: Initialized, Role: FOLLOWER
I20260926 11:38:01.887074 13218 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260926 11:38:01.887236 13218 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:38:01.887338 13218 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:38:01.887470 13218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:38:01.889655 13218 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260926 11:38:01.890048 13218 leader_election.cc:302] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7b64f15a5a7472680728ec2037e0d6e; no voters:
I20260926 11:38:01.890389 13218 leader_election.cc:288] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 11:38:01.890563 13222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 2 FOLLOWER]: Leader election won for term 2
I20260926 11:38:01.890856 13222 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [term 2 LEADER]: Becoming Leader. State: Replica: c7b64f15a5a7472680728ec2037e0d6e, State: Running, Role: LEADER
I20260926 11:38:01.891403 13222 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260926 11:38:01.891829 13218 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:38:01.893041 13223 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c7b64f15a5a7472680728ec2037e0d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260926 11:38:01.893187 13223 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:01.893957 13224 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7b64f15a5a7472680728ec2037e0d6e. Latest consensus state: current_term: 2 leader_uuid: "c7b64f15a5a7472680728ec2037e0d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b64f15a5a7472680728ec2037e0d6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260926 11:38:01.894071 13224 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:01.894371 13231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:38:01.894408 13218 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 11:38:01.900976 13231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:38:01.902096 13231 catalog_manager.cc:1259] Loaded cluster ID: 8f9ceb818d0e4d4b89dcfb15641b861b
I20260926 11:38:01.902235 13231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:38:01.905090 13231 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:38:01.906157 13231 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Loaded TSK: 0
I20260926 11:38:01.907630 13231 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:38:01.983718 13220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:01.984167 13220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:01.984354 13220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:01.989425 13220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:01.989514 13220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:01.989624 13220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:38:01.996554 13220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44937
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:01.998646 13220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:02.000119 13220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:02.011309 13258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:02.011618 13220 server_base.cc:1034] running on GCE node
W20260926 11:38:02.011262 13255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:02.011756 13256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:02.017539 13220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:02.018482 13220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:02.019667 13220 hybrid_clock.cc:648] HybridClock initialized: now 1790422682019630 us; error 69 us; skew 500 ppm
Sep 26 11:38:02 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:02.023252 13220 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:38:02.024495 13220 webserver.cc:460] Webserver started at http://127.0.98.1:44937/ using document root <none> and password file <none>
I20260926 11:38:02.025189 13220 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:02.025315 13220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 11:38:02 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:38:02.048861 13220 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:38:02.053328 13266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:02.055140 13220 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 11:38:02.055384 13220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "4a96d87aa0e740f493098a0becf988a3"
format_stamp: "Formatted at 2026-09-26 11:38:00 on dist-test-slave-4pw5"
server_key: "15f5b5c675839b8dddb36f7920cbb371"
server_key_iv: "51a3acfb22c7ab94d55676fcd4742b16"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:02.056198 13220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:02.085106 13220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:02.088977 13220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:02.089308 13220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:02.090217 13220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:02.091413 13220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:02.091543 13220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:02.091741 13220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:02.091913 13220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:02.105319 13220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34879
I20260926 11:38:02.105398 13379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34879 every 8 connection(s)
I20260926 11:38:02.106902 13220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:38:02.111641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 13220
I20260926 11:38:02.112205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43691
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38799
--webserver_port=34097
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39413/kms with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:38:02 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:02.176383 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:02.112504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:35951)
0926 11:38:02.115701 (+ 3197us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:02.115773 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:02.120920 (+ 5147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:02.127620 (+ 6700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:02.127632 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:02.129852 (+ 2220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:02.135537 (+ 5685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:02.135597 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.140327 (+ 4730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.140334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:02.141905 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:02.141925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.144203 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.144208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:02.144466 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:02.160420 (+ 15954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:02.160446 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:02.168539 (+ 8093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:02.172018 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:02.172031 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:02.172048 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:02.172599 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:02.173133 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:02.173137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:02.173139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:02.173368 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:02.174307 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:02.174314 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:02.174661 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 11:38:02.174865 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 11:38:02.175265 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2895,"thread_start_us":219,"threads_started":1}
I20260926 11:38:02.178681 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:02.114299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35951 (local address 127.0.98.62:38799)
0926 11:38:02.119922 (+ 5623us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:02.119955 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:02.120025 (+ 70us) server_negotiation.cc:407] Connection header received
0926 11:38:02.123613 (+ 3588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:02.123639 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:02.125750 (+ 2111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:02.126245 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:02.136203 (+ 9958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.140008 (+ 3805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.142894 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.143733 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.143924 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:02.168916 (+ 24992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:02.168925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:02.168944 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:02.168991 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:02.171744 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:02.172787 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:02.172794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:02.172801 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:02.172988 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:02.173511 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:02.173514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:02.173516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:02.174066 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:02.174224 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:02.177280 (+ 3056us) server_negotiation.cc:299] Negotiation successful
0926 11:38:02.177631 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5393,"thread_start_us":139,"threads_started":1}
I20260926 11:38:02.182785 13380 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260926 11:38:02.183084 13380 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:02.183816 13380 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
W20260926 11:38:02.186610 13380 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38799 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 11:38:02.186758 13380 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:02.186982 13380 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260926 11:38:02.188248 13380 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:02.188478 13380 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
W20260926 11:38:02.189496 13380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:38:02.279927 13383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:02.280310 13383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:02.280440 13383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:02.285126 13383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:02.285233 13383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:02.285353 13383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:38:02.292573 13383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34097
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:02.293998 13383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:02.295249 13383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:02.306403 13392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:02.306434 13390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:02.306895 13389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:02.307055 13383 server_base.cc:1034] running on GCE node
I20260926 11:38:02.307545 13383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:02.308377 13383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:02.309597 13383 hybrid_clock.cc:648] HybridClock initialized: now 1790422682309553 us; error 57 us; skew 500 ppm
Sep 26 11:38:02 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:02.313488 13383 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:38:02.314952 13383 webserver.cc:460] Webserver started at http://127.0.98.2:34097/ using document root <none> and password file <none>
I20260926 11:38:02.315846 13383 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:02.316035 13383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 11:38:02 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:38:02.344409 13383 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260926 11:38:02.347777 13400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:02.349443 13383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:38:02.349612 13383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "5482f832f0304781bd83015d89084fc2"
format_stamp: "Formatted at 2026-09-26 11:38:01 on dist-test-slave-4pw5"
server_key: "a1258507c15e98d961c7fc03ba6d1dd8"
server_key_iv: "0ab2b8482f1b16f8effb80ef1e75a0c1"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:02.350272 13383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:02.366743 13383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:02.371093 13383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:02.371405 13383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:02.372290 13383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:02.373581 13383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:02.373694 13383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:02.373795 13383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:02.373858 13383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:02.385052 13383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43691
I20260926 11:38:02.386523 13383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:38:02.387492 13513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43691 every 8 connection(s)
I20260926 11:38:02.393981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 13383
I20260926 11:38:02.394601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39875
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38799
--webserver_port=41185
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39413/kms with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:38:02 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:02.407896 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:02.389587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33231 (local address 127.0.98.62:38799)
0926 11:38:02.389767 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:02.389772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:02.390922 (+ 1150us) server_negotiation.cc:407] Connection header received
0926 11:38:02.392350 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:02.392353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:02.392408 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:02.392568 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:02.394139 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.395093 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.397092 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.397461 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.397568 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:02.401186 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:02.401204 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:02.401208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:02.401247 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:02.403658 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:02.404617 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:02.404621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:02.404623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:02.404730 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:02.405336 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:02.405420 (+ 84us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:02.405422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:02.405984 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:02.406098 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:02.407417 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0926 11:38:02.407636 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 11:38:02.410218 13516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:02.390061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:33231)
0926 11:38:02.390698 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:02.390766 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:02.392046 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:02.392742 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:02.392753 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:02.393164 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:02.393929 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:02.393952 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.395296 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.395301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:02.396919 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:02.396934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.397668 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.397673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:02.397905 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:02.398739 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:02.398765 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:02.400915 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:02.403962 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:02.403979 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:02.404003 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:02.404470 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:02.404931 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:02.404936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:02.404941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:02.405196 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:02.406192 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:02.406288 (+ 96us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:02.406879 (+ 591us) client_negotiation.cc:769] Sending connection context
0926 11:38:02.408480 (+ 1601us) client_negotiation.cc:240] Negotiation successful
0926 11:38:02.408961 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":169,"threads_started":1}
I20260926 11:38:02.412223 13514 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260926 11:38:02.412529 13514 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:02.413213 13514 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
W20260926 11:38:02.415289 13514 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38799 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 11:38:02.415445 13514 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:02.415712 13514 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260926 11:38:02.416787 13514 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:02.417016 13514 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
W20260926 11:38:02.418018 13514 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:38:02.561394 13517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:02.561808 13517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:02.561961 13517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:02.567008 13517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:02.567118 13517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:02.567240 13517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:38:02.575528 13517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41185
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:02.577401 13517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:02.578852 13517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:02.589870 13526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:02.589865 13523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:02.589865 13524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:02.590710 13517 server_base.cc:1034] running on GCE node
I20260926 11:38:02.591284 13517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:02.592296 13517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:02.593520 13517 hybrid_clock.cc:648] HybridClock initialized: now 1790422682593483 us; error 52 us; skew 500 ppm
Sep 26 11:38:02 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:02.597241 13517 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:38:02.598578 13517 webserver.cc:460] Webserver started at http://127.0.98.3:41185/ using document root <none> and password file <none>
I20260926 11:38:02.599637 13517 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:02.599813 13517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 11:38:02 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:38:02.640527 13517 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260926 11:38:02.644397 13534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:02.646256 13517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:38:02.646473 13517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd"
format_stamp: "Formatted at 2026-09-26 11:38:01 on dist-test-slave-4pw5"
server_key: "c0a567e1b1c7f02392c45ef04248c0d2"
server_key_iv: "49bcc36f541e0585b4432cee270021b0"
server_key_version: "kuduclusterkey@0"
I20260926 11:38:02.647370 13517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:02.678207 13517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:02.688515 13517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:02.688854 13517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:02.689960 13517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:02.691272 13517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:02.691361 13517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:02.691450 13517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:02.691512 13517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:02.705108 13517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39875
I20260926 11:38:02.705283 13647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39875 every 8 connection(s)
I20260926 11:38:02.706319 13517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:38:02.709333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 13517
Sep 26 11:38:02 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:02.728433 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:02.710564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.0.98.62:38799)
0926 11:38:02.710818 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:02.710822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:02.711267 (+ 445us) server_negotiation.cc:407] Connection header received
0926 11:38:02.711382 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:02.711387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:02.711446 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:02.711547 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:02.712852 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.713953 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.716952 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.717362 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.717494 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:02.719536 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:02.719541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:02.719545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:02.719611 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:02.723502 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:02.725022 (+ 1520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:02.725027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:02.725029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:02.725158 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:02.726278 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:02.726283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:02.726285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:02.726627 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:02.726872 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:02.727820 (+ 948us) server_negotiation.cc:299] Negotiation successful
0926 11:38:02.728104 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260926 11:38:02.739710 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:02.710453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39629 (local address 127.0.98.62:38799)
0926 11:38:02.710634 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:02.710642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:02.712150 (+ 1508us) server_negotiation.cc:407] Connection header received
0926 11:38:02.714967 (+ 2817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:02.714970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:02.715029 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:02.715107 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:02.717807 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.718714 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.721951 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.722360 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.722473 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:02.728786 (+ 6313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:02.728792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:02.728796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:02.728839 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:02.735776 (+ 6937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:02.736766 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:02.736772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:02.736775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:02.736906 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:02.737523 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:02.737528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:02.737530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:02.737967 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:02.738079 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:02.739052 (+ 973us) server_negotiation.cc:299] Negotiation successful
0926 11:38:02.739251 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 11:38:02.742260 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:02.709151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:39629)
0926 11:38:02.711985 (+ 2834us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:02.712054 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:02.714615 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:02.715625 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:02.715637 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:02.716051 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:02.716859 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:02.717637 (+ 778us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.718901 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.718904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:02.721256 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:02.721274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:02.723865 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:02.723870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:02.724180 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:02.725011 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:02.725039 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:02.727473 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:02.735997 (+ 8524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:02.736008 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:02.736035 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:02.736578 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:02.737058 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:02.737062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:02.737064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:02.737361 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:02.738164 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:02.738173 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:02.738677 (+ 504us) client_negotiation.cc:769] Sending connection context
0926 11:38:02.740294 (+ 1617us) client_negotiation.cc:240] Negotiation successful
0926 11:38:02.740843 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2574,"thread_start_us":173,"threads_started":1}
I20260926 11:38:02.745020 13648 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260926 11:38:02.745365 13648 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:02.746031 13648 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
W20260926 11:38:02.748013 13648 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38799 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 11:38:02.748168 13648 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:02.748387 13648 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260926 11:38:02.749420 13648 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:02.749672 13648 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
W20260926 11:38:02.750530 13648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 11:38:03.189836 13380 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:03.190241 13380 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260926 11:38:03.192013 13178 ts_manager.cc:194] Registered new tserver with Master: 4a96d87aa0e740f493098a0becf988a3 (127.0.98.1:34879)
I20260926 11:38:03.193960 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35951
Sep 26 11:38:03 dist-test-slave-4pw5 krb5kdc[12186](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:03.418303 13514 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:03.418661 13514 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260926 11:38:03.419466 13178 ts_manager.cc:194] Registered new tserver with Master: 5482f832f0304781bd83015d89084fc2 (127.0.98.2:43691)
I20260926 11:38:03.420421 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33231
Sep 26 11:38:03 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:38:03.750820 13648 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:03.751168 13648 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260926 11:38:03.752085 13178 ts_manager.cc:194] Registered new tserver with Master: 7b6b7f4deb1a44c98f8993fb18bd24bd (127.0.98.3:39875)
I20260926 11:38:03.752930 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39629
I20260926 11:38:03.757845 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:38:03.770395 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.759928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.0.98.62:38799)
0926 11:38:03.760235 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:03.760241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:03.760515 (+ 274us) server_negotiation.cc:407] Connection header received
0926 11:38:03.760733 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:03.760737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:03.760792 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:03.760921 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:03.762070 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.762980 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.764144 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.764525 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.764644 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.766381 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:03.766386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:03.766389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:03.766422 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:03.768459 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:03.768979 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:03.768983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:03.768985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:03.769086 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:03.769399 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:03.769402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:03.769404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:03.769642 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:03.769795 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:03.770052 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 11:38:03.770191 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":92,"threads_started":1}
I20260926 11:38:03.775091 13178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:38:03.779997 13178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:38:03.800791 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.792331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39875 (local address 127.0.0.1:55714)
0926 11:38:03.794270 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:03.794325 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:03.794482 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:03.795364 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:03.795371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:03.795408 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:03.795550 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.795559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.797356 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.797362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.798801 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.798812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.800183 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.800187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.800316 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.800357 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 11:38:03.800489 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 11:38:03.800625 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1761,"thread_start_us":129,"threads_started":1}
I20260926 11:38:03.801736 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.792447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.0.98.3:39875)
0926 11:38:03.793997 (+ 1550us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:03.794006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:03.794635 (+ 629us) server_negotiation.cc:407] Connection header received
0926 11:38:03.794777 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:03.794782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:03.795013 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:03.795175 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:03.795732 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.797206 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.798946 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.799838 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.799949 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.800117 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:03.801471 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0926 11:38:03.801592 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1429,"thread_start_us":72,"threads_started":1}
I20260926 11:38:03.804579 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.793995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34879 (local address 127.0.0.1:48640)
0926 11:38:03.795320 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:03.795334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:03.795430 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:03.796760 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:03.796764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:03.796784 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:03.796889 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.796896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.798370 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.798374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.801720 (+ 3346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.801731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.804117 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.804121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.804232 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.804262 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 11:38:03.804340 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 11:38:03.804423 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":94,"threads_started":1}
I20260926 11:38:03.805325 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.794400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.0.98.1:34879)
0926 11:38:03.795179 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:03.795187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:03.795525 (+ 338us) server_negotiation.cc:407] Connection header received
0926 11:38:03.795633 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:03.795638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:03.795833 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:03.795970 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:03.797029 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.798217 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.801892 (+ 3675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.802791 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.803037 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.804991 (+ 1954us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:03.805054 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 11:38:03.805165 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":92,"threads_started":1}
I20260926 11:38:03.805284 13564 tablet_service.cc:1467] Processing CreateTablet for tablet d2a88bb5593f4146ad7c1ca352af6f81 (DEFAULT_TABLE table=test-table [id=dd602710bcff48e995b0a8a99d5816a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:03.806447 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a88bb5593f4146ad7c1ca352af6f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:03.808295 13314 tablet_service.cc:1467] Processing CreateTablet for tablet d2a88bb5593f4146ad7c1ca352af6f81 (DEFAULT_TABLE table=test-table [id=dd602710bcff48e995b0a8a99d5816a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:03.809357 13314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a88bb5593f4146ad7c1ca352af6f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:03.809549 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.792445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43691 (local address 127.0.0.1:36212)
0926 11:38:03.794820 (+ 2375us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:03.794836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:03.794930 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:03.800954 (+ 6024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:03.800958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:03.800979 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:03.801078 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.801085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.803628 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.803631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.804987 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.804998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.809005 (+ 4007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.809009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.809117 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.809148 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 11:38:03.809230 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 11:38:03.809317 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2309,"thread_start_us":106,"threads_started":1}
I20260926 11:38:03.810883 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.793234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.0.98.2:43691)
0926 11:38:03.800241 (+ 7007us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:03.800249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:03.800279 (+ 30us) server_negotiation.cc:407] Connection header received
0926 11:38:03.800368 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:03.800374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:03.800603 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:03.800777 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:03.802165 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.803450 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.805139 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.807756 (+ 2617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.807886 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.810538 (+ 2652us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:03.810602 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 11:38:03.810722 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6871,"thread_start_us":99,"threads_started":1}
I20260926 11:38:03.814030 13448 tablet_service.cc:1467] Processing CreateTablet for tablet d2a88bb5593f4146ad7c1ca352af6f81 (DEFAULT_TABLE table=test-table [id=dd602710bcff48e995b0a8a99d5816a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:03.815171 13448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a88bb5593f4146ad7c1ca352af6f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:03.819182 13682 tablet_bootstrap.cc:492] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: Bootstrap starting.
I20260926 11:38:03.822469 13682 tablet_bootstrap.cc:654] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:03.823381 13682 log.cc:824] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:03.824199 13681 tablet_bootstrap.cc:492] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3: Bootstrap starting.
I20260926 11:38:03.826728 13681 tablet_bootstrap.cc:654] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:03.827370 13682 tablet_bootstrap.cc:492] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: No bootstrap required, opened a new log
I20260926 11:38:03.827615 13682 ts_tablet_manager.cc:1397] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 11:38:03.827687 13681 log.cc:824] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:03.828459 13683 tablet_bootstrap.cc:492] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: Bootstrap starting.
I20260926 11:38:03.831722 13682 raft_consensus.cc:348] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } }
I20260926 11:38:03.832027 13683 tablet_bootstrap.cc:654] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:03.832062 13682 raft_consensus.cc:374] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:03.832113 13682 raft_consensus.cc:729] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6b7f4deb1a44c98f8993fb18bd24bd, State: Initialized, Role: FOLLOWER
I20260926 11:38:03.832638 13682 consensus_queue.cc:260] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } }
I20260926 11:38:03.832899 13683 log.cc:824] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:03.833781 13682 ts_tablet_manager.cc:1428] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:38:03.834158 13648 heartbeater.cc:503] Master 127.0.98.62:38799 was elected leader, sending a full tablet report...
I20260926 11:38:03.835240 13681 tablet_bootstrap.cc:492] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3: No bootstrap required, opened a new log
I20260926 11:38:03.835516 13681 ts_tablet_manager.cc:1397] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260926 11:38:03.840026 13683 tablet_bootstrap.cc:492] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: No bootstrap required, opened a new log
I20260926 11:38:03.840252 13683 ts_tablet_manager.cc:1397] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260926 11:38:03.840008 13681 raft_consensus.cc:348] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } }
I20260926 11:38:03.841019 13681 raft_consensus.cc:374] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:03.841099 13681 raft_consensus.cc:729] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a96d87aa0e740f493098a0becf988a3, State: Initialized, Role: FOLLOWER
I20260926 11:38:03.841688 13681 consensus_queue.cc:260] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } }
I20260926 11:38:03.842857 13681 ts_tablet_manager.cc:1428] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260926 11:38:03.843276 13380 heartbeater.cc:503] Master 127.0.98.62:38799 was elected leader, sending a full tablet report...
I20260926 11:38:03.845611 13683 raft_consensus.cc:348] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } }
I20260926 11:38:03.845928 13683 raft_consensus.cc:374] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:03.845992 13683 raft_consensus.cc:729] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5482f832f0304781bd83015d89084fc2, State: Initialized, Role: FOLLOWER
I20260926 11:38:03.846499 13683 consensus_queue.cc:260] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } }
I20260926 11:38:03.847677 13683 ts_tablet_manager.cc:1428] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 11:38:03.848040 13514 heartbeater.cc:503] Master 127.0.98.62:38799 was elected leader, sending a full tablet report...
W20260926 11:38:03.869473 13381 tablet.cc:2369] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:38:03.889320 13515 tablet.cc:2369] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:38:03.950699 13686 raft_consensus.cc:482] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:03.950946 13686 raft_consensus.cc:504] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } }
Sep 26 11:38:03 dist-test-slave-4pw5 krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:38:03.952502 13686 leader_election.cc:288] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5482f832f0304781bd83015d89084fc2 (127.0.98.2:43691), 4a96d87aa0e740f493098a0becf988a3 (127.0.98.1:34879)
W20260926 11:38:03.958494 13649 tablet.cc:2369] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:38:03.960534 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.953342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38039 (local address 127.0.98.2:43691)
0926 11:38:03.953487 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:03.953492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:03.954524 (+ 1032us) server_negotiation.cc:407] Connection header received
0926 11:38:03.954748 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:03.954752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:03.954805 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:03.954855 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:03.955307 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.956102 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.958215 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.958900 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.958984 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.959055 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:03.960272 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0926 11:38:03.960373 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:38:03.962827 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.954067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37307 (local address 127.0.98.1:34879)
0926 11:38:03.954184 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:03.954189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:03.957971 (+ 3782us) server_negotiation.cc:407] Connection header received
0926 11:38:03.958207 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:03.958212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:03.958282 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:03.958382 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:03.958886 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.959691 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.961522 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.962115 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.962232 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.962301 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:03.962615 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 11:38:03.962699 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:38:03.963467 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.953708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34879 (local address 127.0.98.3:37307)
0926 11:38:03.957882 (+ 4174us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:03.957899 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:03.958044 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:03.958508 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:03.958514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:03.958537 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:03.958670 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.958689 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.960021 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.960025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.961325 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.961338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.962367 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.962370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.962472 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.962496 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:38:03.963211 (+ 715us) client_negotiation.cc:240] Negotiation successful
0926 11:38:03.963301 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4106,"thread_start_us":87,"threads_started":1}
I20260926 11:38:03.963886 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:03.952912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43691 (local address 127.0.98.3:38039)
0926 11:38:03.954384 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:03.954408 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:03.954615 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:03.955014 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:03.955017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:03.955035 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:03.955176 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.955182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.956252 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.956256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.957609 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:03.957621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:03.959440 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:03.959446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:03.959776 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:03.959804 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 11:38:03.963633 (+ 3829us) client_negotiation.cc:240] Negotiation successful
0926 11:38:03.963735 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"thread_start_us":127,"threads_started":1}
I20260926 11:38:03.971222 13461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a88bb5593f4146ad7c1ca352af6f81" candidate_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5482f832f0304781bd83015d89084fc2" is_pre_election: true
I20260926 11:38:03.971212 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a88bb5593f4146ad7c1ca352af6f81" candidate_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a96d87aa0e740f493098a0becf988a3" is_pre_election: true
I20260926 11:38:03.971889 13334 raft_consensus.cc:2454] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6b7f4deb1a44c98f8993fb18bd24bd in term 0.
I20260926 11:38:03.971889 13461 raft_consensus.cc:2454] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6b7f4deb1a44c98f8993fb18bd24bd in term 0.
I20260926 11:38:03.972651 13538 leader_election.cc:302] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a96d87aa0e740f493098a0becf988a3, 7b6b7f4deb1a44c98f8993fb18bd24bd; no voters:
I20260926 11:38:03.973253 13686 raft_consensus.cc:2781] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:38:03.973364 13686 raft_consensus.cc:482] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:38:03.973423 13686 raft_consensus.cc:3037] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:03.974992 13686 raft_consensus.cc:504] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } }
I20260926 11:38:03.975476 13686 leader_election.cc:288] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [CANDIDATE]: Term 1 election: Requested vote from peers 5482f832f0304781bd83015d89084fc2 (127.0.98.2:43691), 4a96d87aa0e740f493098a0becf988a3 (127.0.98.1:34879)
I20260926 11:38:03.976418 13461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a88bb5593f4146ad7c1ca352af6f81" candidate_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5482f832f0304781bd83015d89084fc2"
I20260926 11:38:03.976603 13461 raft_consensus.cc:3037] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:03.976966 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a88bb5593f4146ad7c1ca352af6f81" candidate_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a96d87aa0e740f493098a0becf988a3"
I20260926 11:38:03.977125 13334 raft_consensus.cc:3037] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:03.978139 13461 raft_consensus.cc:2454] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6b7f4deb1a44c98f8993fb18bd24bd in term 1.
I20260926 11:38:03.978528 13334 raft_consensus.cc:2454] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6b7f4deb1a44c98f8993fb18bd24bd in term 1.
I20260926 11:38:03.978556 13537 leader_election.cc:302] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5482f832f0304781bd83015d89084fc2, 7b6b7f4deb1a44c98f8993fb18bd24bd; no voters:
I20260926 11:38:03.979079 13686 raft_consensus.cc:2781] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:38:03.979463 13686 raft_consensus.cc:686] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 1 LEADER]: Becoming Leader. State: Replica: 7b6b7f4deb1a44c98f8993fb18bd24bd, State: Running, Role: LEADER
I20260926 11:38:03.979928 13686 consensus_queue.cc:237] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } }
I20260926 11:38:03.984004 13178 catalog_manager.cc:5495] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b6b7f4deb1a44c98f8993fb18bd24bd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6b7f4deb1a44c98f8993fb18bd24bd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } health_report { overall_health: UNKNOWN } } }
I20260926 11:38:04.038281 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:04.033344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.0.98.3:39875)
0926 11:38:04.033523 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:04.033528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:04.033546 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:38:04.033645 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:04.033649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:04.033739 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:04.033873 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:38:04.034167 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:04.035028 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:04.036301 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:04.036684 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:04.036819 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:04.037184 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:38:04.037823 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:38:04.037955 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:04.038044 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 11:38:04.038117 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 11:38:04.047799 13334 raft_consensus.cc:1261] T d2a88bb5593f4146ad7c1ca352af6f81 P 4a96d87aa0e740f493098a0becf988a3 [term 1 FOLLOWER]: Refusing update from remote peer 7b6b7f4deb1a44c98f8993fb18bd24bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:04.048732 13461 raft_consensus.cc:1261] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [term 1 FOLLOWER]: Refusing update from remote peer 7b6b7f4deb1a44c98f8993fb18bd24bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:04.050004 13693 consensus_queue.cc:1035] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5482f832f0304781bd83015d89084fc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:04.054279 13693 consensus_queue.cc:1035] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a96d87aa0e740f493098a0becf988a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:04.072309 13697 mvcc.cc:205] Tried to move back new op lower bound from 7333571313849344000 to 7333571313586130944. Current Snapshot: MvccSnapshot[applied={T|T < 7333571313849344000}]
I20260926 11:38:04.077284 13698 mvcc.cc:205] Tried to move back new op lower bound from 7333571313849344000 to 7333571313586130944. Current Snapshot: MvccSnapshot[applied={T|T < 7333571313849344000}]
I20260926 11:38:04.079906 13699 mvcc.cc:205] Tried to move back new op lower bound from 7333571313849344000 to 7333571313586130944. Current Snapshot: MvccSnapshot[applied={T|T < 7333571313849344000}]
I20260926 11:38:04.092677 13182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:04.092921 13182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:04.096282 13182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7b64f15a5a7472680728ec2037e0d6e: Sending DeleteTablet for 3 replicas of tablet d2a88bb5593f4146ad7c1ca352af6f81
I20260926 11:38:04.097678 13448 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a88bb5593f4146ad7c1ca352af6f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36212
I20260926 11:38:04.098050 13314 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a88bb5593f4146ad7c1ca352af6f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48640
I20260926 11:38:04.098946 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a88bb5593f4146ad7c1ca352af6f81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55714
I20260926 11:38:04.100116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 13220
I20260926 11:38:04.111706 13708 tablet_replica.cc:330] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: stopping tablet replica
I20260926 11:38:04.112179 13708 raft_consensus.cc:2229] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:38:04.112576 13706 tablet_replica.cc:330] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: stopping tablet replica
I20260926 11:38:04.112737 13708 raft_consensus.cc:2258] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:04.112913 13706 raft_consensus.cc:2229] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 11:38:04.113256 13169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34879 (error 108)
I20260926 11:38:04.113478 13706 raft_consensus.cc:2258] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:04.115197 13706 ts_tablet_manager.cc:1905] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:04.115198 13708 ts_tablet_manager.cc:1905] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:04.118086 13706 ts_tablet_manager.cc:1918] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:04.118086 13708 ts_tablet_manager.cc:1918] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:04.118202 13708 log.cc:1196] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/d2a88bb5593f4146ad7c1ca352af6f81
I20260926 11:38:04.118201 13706 log.cc:1196] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/d2a88bb5593f4146ad7c1ca352af6f81
I20260926 11:38:04.118551 13708 ts_tablet_manager.cc:1939] T d2a88bb5593f4146ad7c1ca352af6f81 P 7b6b7f4deb1a44c98f8993fb18bd24bd: Deleting consensus metadata
I20260926 11:38:04.118551 13706 ts_tablet_manager.cc:1939] T d2a88bb5593f4146ad7c1ca352af6f81 P 5482f832f0304781bd83015d89084fc2: Deleting consensus metadata
I20260926 11:38:04.119915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 13383
I20260926 11:38:04.124475 13167 catalog_manager.cc:4841] TS 7b6b7f4deb1a44c98f8993fb18bd24bd (127.0.98.3:39875): tablet d2a88bb5593f4146ad7c1ca352af6f81 (table test-table [id=dd602710bcff48e995b0a8a99d5816a6]) successfully deleted
I20260926 11:38:04.125764 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:04.125374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34879 (local address 127.0.0.1:48654)
0926 11:38:04.125543 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:04.125652 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 11:38:04.125968 13169 catalog_manager.cc:4584] TS 4a96d87aa0e740f493098a0becf988a3 (127.0.98.1:34879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2a88bb5593f4146ad7c1ca352af6f81: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34879: connect: Connection refused (error 111)
W20260926 11:38:04.144311 13679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:38:04.143359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43691 (local address 127.0.0.1:36224)
0926 11:38:04.143759 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:04.143778 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:04.143902 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:04.144174 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127}
I20260926 11:38:04.152829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 13517
I20260926 11:38:04.161932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 13149
I20260926 11:38:04.170331 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 11:38:04.506431 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 11:38:04.506556 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 11:38:04.852708 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 11:38:04.852 UTC [12216] LOG: received smart shutdown request
2026-09-26 11:38:04.857 UTC [12216] LOG: background worker "logical replication launcher" (PID 12225) exited with exit code 1
2026-09-26 11:38:04.858 UTC [12220] LOG: shutting down
2026-09-26 11:38:05.102 UTC [12216] LOG: database system is shut down
2026-09-26T11:38:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53597 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:38:05 dist-test-slave-4pw5 krb5kdc[13718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:38:05 dist-test-slave-4pw5 krb5kdc[13718](info): set up 2 sockets
Sep 26 11:38:05 dist-test-slave-4pw5 krb5kdc[13718](info): commencing operation
krb5kdc: starting...
W20260926 11:38:06.364158 392 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:38:06 dist-test-slave-4pw5 krb5kdc[13718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:38:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:38:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:38:06.504331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41613,127.0.98.61:40039,127.0.98.60:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:06.633888 13731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:06.634323 13731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:06.634414 13731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:06.639437 13731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:06.639619 13731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:06.639673 13731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:06.639708 13731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:06.639751 13731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:06.646025 13731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41613,127.0.98.61:40039,127.0.98.60:40481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:06.647334 13731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:06.648499 13731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:06.657845 13736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:06.657845 13737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:06.657892 13731 server_base.cc:1034] running on GCE node
W20260926 11:38:06.658102 13739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:06.658819 13731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:06.660158 13731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:06.661407 13731 hybrid_clock.cc:648] HybridClock initialized: now 1790422686661384 us; error 39 us; skew 500 ppm
Sep 26 11:38:06 dist-test-slave-4pw5 krb5kdc[13718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:06.664924 13731 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:38:06.665995 13731 webserver.cc:460] Webserver started at http://127.0.98.62:34959/ using document root <none> and password file <none>
I20260926 11:38:06.666585 13731 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:06.666651 13731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:06.666831 13731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:06.669293 13731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "0f41f17b8f8241308f4ee88885ff8804"
format_stamp: "Formatted at 2026-09-26 11:38:06 on dist-test-slave-4pw5"
server_key: "a315868765ef10f61f34a9fe3210d09c"
server_key_iv: "ba2d3d31e350749c3d43d40852100b0f"
server_key_version: "encryptionkey@0"
I20260926 11:38:06.669867 13731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f41f17b8f8241308f4ee88885ff8804"
format_stamp: "Formatted at 2026-09-26 11:38:06 on dist-test-slave-4pw5"
server_key: "a315868765ef10f61f34a9fe3210d09c"
server_key_iv: "ba2d3d31e350749c3d43d40852100b0f"
server_key_version: "encryptionkey@0"
I20260926 11:38:06.674062 13731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:38:06.677309 13746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:06.678799 13731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:38:06.678982 13731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "0f41f17b8f8241308f4ee88885ff8804"
format_stamp: "Formatted at 2026-09-26 11:38:06 on dist-test-slave-4pw5"
server_key: "a315868765ef10f61f34a9fe3210d09c"
server_key_iv: "ba2d3d31e350749c3d43d40852100b0f"
server_key_version: "encryptionkey@0"
I20260926 11:38:06.679127 13731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:06.706974 13731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:06.710621 13731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:06.710922 13731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:06.719010 13731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41613
I20260926 11:38:06.719041 13798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41613 every 8 connection(s)
I20260926 11:38:06.720062 13731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:38:06.722548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 13731
I20260926 11:38:06.722671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:38:06.722995 392 external_mini_cluster.cc:1428] Setting key 893facad4fc53adc351e83d4183afab6
I20260926 11:38:06.723408 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:06.729216 13799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 11:38:06 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:06.747555 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.732422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41613 (local address 127.0.0.1:44198)
0926 11:38:06.733082 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.733135 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:06.734117 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:06.734866 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:06.734872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:06.735317 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:06.735638 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:06.735654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.736808 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.736811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:06.737563 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:06.737573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.738257 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.738261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:06.738373 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:06.739204 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:06.739223 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:06.741208 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:06.743975 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:06.743984 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:06.743993 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:06.744254 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:06.744760 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:06.744763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:06.744765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:06.744891 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:06.745736 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:06.745746 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:06.745920 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 11:38:06.746089 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 11:38:06.746606 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":127,"spinlock_wait_cycles":2816,"thread_start_us":162,"threads_started":1}
I20260926 11:38:06.747562 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.732401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.0.98.62:41613)
0926 11:38:06.732990 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:06.733010 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:06.733258 (+ 248us) server_negotiation.cc:407] Connection header received
0926 11:38:06.734446 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:06.734452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:06.734688 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:06.734870 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:06.735781 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.736675 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.737708 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.738092 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.738277 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:06.741398 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:06.741407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:06.741423 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:06.741455 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:06.743806 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:06.744445 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:06.744451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:06.744452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:06.744582 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:06.745066 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:06.745069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:06.745070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:06.745541 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:06.745744 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:06.746200 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 11:38:06.746515 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"spinlock_wait_cycles":55168,"thread_start_us":179,"threads_started":1}
I20260926 11:38:06.750471 13799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:38:06.751421 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.751103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40039 (local address 127.0.0.1:55482)
0926 11:38:06.751246 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.751345 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 11:38:06.751660 13750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
I20260926 11:38:06.752763 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.752538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40039 (local address 127.0.0.1:55494)
0926 11:38:06.752650 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.752705 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 11:38:06.752985 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40039: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 11:38:06.756414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40039
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41613,127.0.98.61:40039,127.0.98.60:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:38:06.802263 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } attempt: 1
I20260926 11:38:06.803437 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.802998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40039 (local address 127.0.0.1:55496)
0926 11:38:06.803195 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.803270 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 11:38:06.804265 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.804041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40039 (local address 127.0.0.1:55500)
0926 11:38:06.804156 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.804208 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 11:38:06.804383 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40039: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
I20260926 11:38:06.872574 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } attempt: 2
I20260926 11:38:06.873643 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.873252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40039 (local address 127.0.0.1:55514)
0926 11:38:06.873463 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.873544 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 11:38:06.874590 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.874314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40039 (local address 127.0.0.1:55516)
0926 11:38:06.874477 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.874529 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 11:38:06.874747 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40039: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40039: connect: Connection refused (error 111)
W20260926 11:38:06.879536 13806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:06.879987 13806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:06.880098 13806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:06.884795 13806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:06.884893 13806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:06.884931 13806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:06.885003 13806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:06.885037 13806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:06.890950 13806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41613,127.0.98.61:40039,127.0.98.60:40481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:06.892295 13806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:06.893507 13806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:06.902664 13811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:06.902689 13814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:06.902849 13806 server_base.cc:1034] running on GCE node
W20260926 11:38:06.902668 13812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:06.904033 13806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:06.905294 13806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:06.906513 13806 hybrid_clock.cc:648] HybridClock initialized: now 1790422686906501 us; error 34 us; skew 500 ppm
Sep 26 11:38:06 dist-test-slave-4pw5 krb5kdc[13718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:06.910177 13806 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 11:38:06.911783 13806 webserver.cc:460] Webserver started at http://127.0.98.61:43957/ using document root <none> and password file <none>
I20260926 11:38:06.912539 13806 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:06.912653 13806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:06.912926 13806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:06.915396 13806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/data/instance:
uuid: "bdefbb5127c04944a8d5980dd95dcacd"
format_stamp: "Formatted at 2026-09-26 11:38:06 on dist-test-slave-4pw5"
server_key: "ba219afc821438f71318f60a5f5c37b1"
server_key_iv: "dcc906479070e499ea80d7b4d7470a6c"
server_key_version: "encryptionkey@0"
I20260926 11:38:06.916175 13806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/wal/instance:
uuid: "bdefbb5127c04944a8d5980dd95dcacd"
format_stamp: "Formatted at 2026-09-26 11:38:06 on dist-test-slave-4pw5"
server_key: "ba219afc821438f71318f60a5f5c37b1"
server_key_iv: "dcc906479070e499ea80d7b4d7470a6c"
server_key_version: "encryptionkey@0"
I20260926 11:38:06.921666 13806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 11:38:06.925459 13821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:06.927116 13806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:38:06.927345 13806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/wal
uuid: "bdefbb5127c04944a8d5980dd95dcacd"
format_stamp: "Formatted at 2026-09-26 11:38:06 on dist-test-slave-4pw5"
server_key: "ba219afc821438f71318f60a5f5c37b1"
server_key_iv: "dcc906479070e499ea80d7b4d7470a6c"
server_key_version: "encryptionkey@0"
I20260926 11:38:06.927526 13806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:06.953596 13806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:06.957499 13806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:06.957737 13806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:06.965958 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } attempt: 3
I20260926 11:38:06.966140 13806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40039
I20260926 11:38:06.966171 13873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40039 every 8 connection(s)
I20260926 11:38:06.967741 13806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:38:06.970906 13875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:06.974614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 13806
I20260926 11:38:06.974753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-1/wal/instance
I20260926 11:38:06.975121 392 external_mini_cluster.cc:1428] Setting key 900bb0d6a83e12dd3932dc2075761d9b
Sep 26 11:38:06 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:38:06.976079 13875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:38:06.980413 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.966535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40039 (local address 127.0.0.1:55518)
0926 11:38:06.966701 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.966723 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:06.967042 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:06.969827 (+ 2785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:06.969830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:06.970054 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:06.970203 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:06.970212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.971662 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.971667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:06.972488 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:06.972501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.973284 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.973288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:06.973400 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:06.973973 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:06.973989 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:06.975642 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:06.978056 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:06.978061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:06.978065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:06.978370 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:06.978790 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:06.978793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:06.978795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:06.978917 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:06.979720 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:06.979726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:06.979867 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 11:38:06.979995 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 11:38:06.980177 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 11:38:06.981695 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.967556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.0.98.61:40039)
0926 11:38:06.968145 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:06.968161 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:06.968209 (+ 48us) server_negotiation.cc:407] Connection header received
0926 11:38:06.969119 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:06.969137 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:06.969505 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:06.969805 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:06.970382 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.971477 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.972645 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.973140 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.973331 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:06.975888 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:06.975897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:06.975918 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:06.975960 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:06.977900 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:06.978506 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:06.978514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:06.978523 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:06.978672 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:06.979031 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:06.979036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:06.979038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:06.979595 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:06.979729 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:06.980358 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 11:38:06.980722 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":164,"threads_started":1}
I20260926 11:38:06.984234 13799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 11:38:06.985124 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.984786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54598)
0926 11:38:06.984969 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.985037 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 11:38:06.985916 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.985638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54612)
0926 11:38:06.985816 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.985864 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260926 11:38:06.986153 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40481: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 11:38:06 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:06.994025 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.978407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.0.98.62:41613)
0926 11:38:06.978545 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:06.978549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:06.978891 (+ 342us) server_negotiation.cc:407] Connection header received
0926 11:38:06.979091 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:06.979094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:06.979141 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:06.979192 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:06.980181 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.981393 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.982859 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.983308 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.983397 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:06.989632 (+ 6235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:06.989636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:06.989638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:06.989672 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:06.991868 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:06.992501 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:06.992504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:06.992506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:06.992591 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:06.992970 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:06.992974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:06.992980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:06.993188 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:06.993303 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:06.993705 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 11:38:06.993847 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:38:06.994042 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.978345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41613 (local address 127.0.0.1:44202)
0926 11:38:06.978775 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.978828 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:06.978982 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:06.979351 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:06.979357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:06.979831 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:06.980007 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:06.980016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.981550 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.981556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:06.982683 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:06.982697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.983459 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.983462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:06.983611 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:06.984277 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:06.984293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:06.989347 (+ 5054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:06.992031 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:06.992046 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:06.992051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:06.992331 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:06.992711 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:06.992715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:06.992717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:06.992813 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:06.993320 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:06.993329 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:06.993464 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 11:38:06.993628 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 11:38:06.993858 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":141,"threads_started":1}
I20260926 11:38:06.995282 13875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:38:06 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:38:07.005292 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.995969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40039 (local address 127.0.0.1:55530)
0926 11:38:06.996174 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:06.996233 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:06.996335 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:06.996588 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:06.996591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:06.996774 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:06.996900 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:06.996906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.997736 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.997743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:06.998341 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:06.998349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.998950 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.998953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:06.999117 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:06.999726 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:06.999740 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:07.001233 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:07.003229 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.003232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.003234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.003726 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.004088 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.004091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.004092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.004186 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.004743 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:07.004746 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:07.004863 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 11:38:07.004971 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 11:38:07.005119 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":4}
I20260926 11:38:07.005330 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:06.996154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.0.98.61:40039)
0926 11:38:06.996346 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:06.996350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:06.996367 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:38:06.996436 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:06.996440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:06.996485 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:06.996577 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:06.997022 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.997618 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.998503 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:06.998827 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:06.998951 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.001426 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.001430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.001431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.001455 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.003099 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.003864 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.003869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.003871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.003961 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.004348 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.004351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.004353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.004627 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.004717 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.005049 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.005194 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":65}
I20260926 11:38:07.006301 13875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 11:38:07.007196 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.006839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54618)
0926 11:38:07.006996 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.007118 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 11:38:07.008019 13823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
I20260926 11:38:07.008074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40481
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41613,127.0.98.61:40039,127.0.98.60:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:38:07.009083 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.008827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54630)
0926 11:38:07.008965 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.009014 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 11:38:07.009256 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40481: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
I20260926 11:38:07.017365 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } attempt: 1
I20260926 11:38:07.018460 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.018106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54644)
0926 11:38:07.018309 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.018372 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 11:38:07.019418 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.019133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54652)
0926 11:38:07.019284 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.019346 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 11:38:07.019690 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40481: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
I20260926 11:38:07.058565 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } attempt: 1
I20260926 11:38:07.059536 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.059174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54660)
0926 11:38:07.059361 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.059440 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 11:38:07.060382 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.060136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54668)
0926 11:38:07.060252 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.060309 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 11:38:07.060642 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40481: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
I20260926 11:38:07.094906 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } attempt: 2
I20260926 11:38:07.095892 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.095532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54678)
0926 11:38:07.095745 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.095810 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 11:38:07.096601 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.096394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54680)
0926 11:38:07.096502 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.096551 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 11:38:07.096725 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40481: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
I20260926 11:38:07.128839 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } attempt: 2
I20260926 11:38:07.129765 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.129449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54694)
0926 11:38:07.129621 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.129685 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 11:38:07.130520 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.130307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54700)
0926 11:38:07.130417 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.130467 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 11:38:07.130653 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40481: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
W20260926 11:38:07.130802 13881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:07.131130 13881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:07.131238 13881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:07.135758 13881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:07.135864 13881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:07.135908 13881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:07.135958 13881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:07.136016 13881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:07.141770 13881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41613,127.0.98.61:40039,127.0.98.60:40481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:07.143023 13881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:07.144161 13881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:07.153714 13886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:07.153726 13889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:07.153718 13887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:07.153981 13881 server_base.cc:1034] running on GCE node
I20260926 11:38:07.154753 13881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:07.156075 13881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:07.157289 13881 hybrid_clock.cc:648] HybridClock initialized: now 1790422687157270 us; error 42 us; skew 500 ppm
Sep 26 11:38:07 dist-test-slave-4pw5 krb5kdc[13718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:07.160817 13881 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 11:38:07.161877 13881 webserver.cc:460] Webserver started at http://127.0.98.60:39251/ using document root <none> and password file <none>
I20260926 11:38:07.162487 13881 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:07.162585 13881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:07.162839 13881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:07.165349 13881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/data/instance:
uuid: "c0d0d1b8874e4f6fa936101c863a7b5d"
format_stamp: "Formatted at 2026-09-26 11:38:07 on dist-test-slave-4pw5"
server_key: "643968ef390900f3dbbd845c2f593965"
server_key_iv: "0d7d2e5a3c652ff619dbf379b9799ac8"
server_key_version: "encryptionkey@0"
I20260926 11:38:07.166023 13881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/wal/instance:
uuid: "c0d0d1b8874e4f6fa936101c863a7b5d"
format_stamp: "Formatted at 2026-09-26 11:38:07 on dist-test-slave-4pw5"
server_key: "643968ef390900f3dbbd845c2f593965"
server_key_iv: "0d7d2e5a3c652ff619dbf379b9799ac8"
server_key_version: "encryptionkey@0"
I20260926 11:38:07.170202 13881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:38:07.173146 13896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:07.174645 13881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:38:07.174855 13881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/wal
uuid: "c0d0d1b8874e4f6fa936101c863a7b5d"
format_stamp: "Formatted at 2026-09-26 11:38:07 on dist-test-slave-4pw5"
server_key: "643968ef390900f3dbbd845c2f593965"
server_key_iv: "0d7d2e5a3c652ff619dbf379b9799ac8"
server_key_version: "encryptionkey@0"
I20260926 11:38:07.175040 13881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:07.188997 13881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:07.192921 13881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:07.193157 13881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:07.195887 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } attempt: 3
I20260926 11:38:07.196949 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.196575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54714)
0926 11:38:07.196751 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.196824 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 11:38:07.199030 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.197862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54722)
0926 11:38:07.198849 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.198924 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":883}
W20260926 11:38:07.199225 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40481: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40481: connect: Connection refused (error 111)
I20260926 11:38:07.202544 13881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40481
I20260926 11:38:07.202595 13948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40481 every 8 connection(s)
I20260926 11:38:07.203717 13881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
I20260926 11:38:07.206522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 13881
I20260926 11:38:07.206707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422511662684-392-0/minicluster-data/master-2/wal/instance
I20260926 11:38:07.207183 392 external_mini_cluster.cc:1428] Setting key 4e1342c513232ad9f197ae760573134f
I20260926 11:38:07.207284 13949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:07.213055 13949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:38:07 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:38:07 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:07.221835 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } attempt: 3
I20260926 11:38:07.225847 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.208841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.98.60:40481)
0926 11:38:07.209356 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.209375 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.209417 (+ 42us) server_negotiation.cc:407] Connection header received
0926 11:38:07.210427 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.210455 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.210910 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.211325 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.212290 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.213799 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.214733 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.215435 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.215634 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.218587 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.218593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.218602 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.218634 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.221064 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.221677 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.221683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.221690 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.221842 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.222567 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.222570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.222572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.223240 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.223554 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.224305 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.224700 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":156,"threads_started":1}
I20260926 11:38:07.226048 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.215121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.0.98.62:41613)
0926 11:38:07.215269 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.215275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.215662 (+ 387us) server_negotiation.cc:407] Connection header received
0926 11:38:07.215897 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.215900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.215945 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.216062 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.217198 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.217743 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.218784 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.219179 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.219339 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.221879 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.221884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.221887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.221920 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.224010 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.224560 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.224563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.224567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.224670 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.225014 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.225018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.225019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.225241 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.225350 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.225719 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.225847 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:38:07.226488 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.214991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41613 (local address 127.0.0.1:44208)
0926 11:38:07.215484 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.215560 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:07.215766 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:07.216141 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:07.216145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:07.216802 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:07.217013 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.217021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.217902 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.217905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.218642 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.218652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.219341 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.219345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.219475 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.219993 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:07.220005 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:07.221671 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:07.224144 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.224151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.224157 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.224442 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.224773 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.224776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.224778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.224866 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.225366 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:07.225372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:07.225494 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 11:38:07.225638 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 11:38:07.225810 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":171,"threads_started":1}
I20260926 11:38:07.227962 13949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:38:07 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:38:07.233347 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.222397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54746)
0926 11:38:07.222579 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.222594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:07.222675 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:07.223173 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:07.223176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:07.223501 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:07.223668 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.223680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.224626 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.224631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.225406 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.225414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.226446 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.226450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.226543 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.227602 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:07.227619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:07.229229 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:07.231460 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.231464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.231466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.231757 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.232071 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.232075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.232077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.232204 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.232625 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:07.232630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:07.232844 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 11:38:07.232954 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 11:38:07.233122 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":55}
I20260926 11:38:07.233385 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.222578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.0.98.60:40481)
0926 11:38:07.222923 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.222930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.222942 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:38:07.222973 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.222976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.223046 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.223146 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.223816 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.224469 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.225521 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.225850 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.226442 (+ 592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.229411 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.229415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.229418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.229442 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.231356 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.231867 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.231870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.231872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.231960 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.232314 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.232317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.232318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.232518 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.232675 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.233053 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.233189 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
Sep 26 11:38:07 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 11:38:07 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:07.238087 13875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd: Bootstrap starting.
I20260926 11:38:07.239822 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.228974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.0.98.61:40039)
0926 11:38:07.229107 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.229112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.229550 (+ 438us) server_negotiation.cc:407] Connection header received
0926 11:38:07.229644 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.229647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.229700 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.229779 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.230392 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.231154 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.232228 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.232586 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.232682 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.235849 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.235853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.235856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.235886 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.237692 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.238219 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.238224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.238225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.238312 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.238606 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.238609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.238611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.238904 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.239015 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.239449 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.239633 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":872,"server-negotiator.queue_time_us":66}
I20260926 11:38:07.239856 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.228617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40039 (local address 127.0.0.1:55534)
0926 11:38:07.229466 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.229480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:07.229565 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:07.229884 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:07.229887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:07.230092 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:07.230234 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.230241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.231318 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.231322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.232111 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.232119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.232765 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.232768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.232859 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.233302 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:07.233315 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:07.235671 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:07.237892 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.237895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.237897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.238110 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.238414 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.238418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.238420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.238512 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.238999 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:07.239002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:07.239121 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 11:38:07.239245 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 11:38:07.239717 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772}
I20260926 11:38:07.240651 13949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:38:07.241446 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.228856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.0.98.62:41613)
0926 11:38:07.229107 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.229112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.229127 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:38:07.229164 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.229167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.229213 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.229286 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.230209 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.230935 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.232060 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.232380 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.232499 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.235946 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.235953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.235959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.235998 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.238395 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.238922 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.238927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.238928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.239020 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.239511 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.239515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.239516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.239809 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.240656 (+ 847us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.241120 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.241260 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260926 11:38:07.241722 13875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:07.242756 13875 log.cc:824] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:07.246018 13875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd: No bootstrap required, opened a new log
Sep 26 11:38:07 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:38:07 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:38:07.250733 13875 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } }
I20260926 11:38:07.250994 13875 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:07.251056 13875 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdefbb5127c04944a8d5980dd95dcacd, State: Initialized, Role: FOLLOWER
I20260926 11:38:07.251376 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.241357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.0.98.60:40481)
0926 11:38:07.241457 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.241463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.241700 (+ 237us) server_negotiation.cc:407] Connection header received
0926 11:38:07.241726 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.241731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.241785 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.241854 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.242323 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.243133 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.244538 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.244876 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.245040 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.247322 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.247327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.247330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.247364 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.249146 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.249674 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.249677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.249679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.249763 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.250019 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.250022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.250023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.250236 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.250336 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.251044 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.251215 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":32}
I20260926 11:38:07.251686 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.241160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54762)
0926 11:38:07.241527 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.241542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:07.241627 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:07.241890 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:07.241894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:07.242097 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:07.242201 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.242209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.243254 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.243257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.244377 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.244392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.245043 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.245047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.245134 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.245739 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:07.245764 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:07.247153 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:07.249292 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.249297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.249299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.249558 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.249854 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.249857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.249859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.249935 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.250368 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:07.250375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:07.250809 (+ 434us) client_negotiation.cc:769] Sending connection context
0926 11:38:07.250955 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 11:38:07.251135 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315}
I20260926 11:38:07.251634 13875 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } }
I20260926 11:38:07.252271 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:07.252444 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:38:07.253258 13875 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:38:07.254894 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.242909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.0.98.61:40039)
0926 11:38:07.243022 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.243027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.243110 (+ 83us) server_negotiation.cc:407] Connection header received
0926 11:38:07.243269 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.243272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.243538 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.243693 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.244770 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.245470 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.246527 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.246837 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.246936 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.250102 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.250109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.250112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.250148 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.252661 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.253204 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.253210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.253214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.253314 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.253647 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.253651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.253653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.253866 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.254069 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.254630 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.254752 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 11:38:07.255218 13949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d: Bootstrap starting.
I20260926 11:38:07.258500 13949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:07.258822 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:38:07.259402 13949 log.cc:824] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d: Log is configured to *not* fsync() on all Append() calls
W20260926 11:38:07.259426 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:38:07.259511 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:38:07.257401 13806 master_runner.cc:186] Error getting master registration for 127.0.98.62:41613: OK, instance_id { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" instance_seqno: 1790422686716731 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:38:07.260978 13806 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:38:07.261943 13949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d: No bootstrap required, opened a new log
I20260926 11:38:07.265681 13949 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } }
I20260926 11:38:07.265933 13949 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:07.266006 13949 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d0d1b8874e4f6fa936101c863a7b5d, State: Initialized, Role: FOLLOWER
I20260926 11:38:07.266495 13949 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } }
I20260926 11:38:07.267822 13949 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [sys.catalog]: configured and running, proceeding with master startup.
W20260926 11:38:07.274637 13988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:38:07.274746 13988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:38:07.275121 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.261460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44222 (local address 127.0.98.62:41613)
0926 11:38:07.261598 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.261603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.264836 (+ 3233us) server_negotiation.cc:407] Connection header received
0926 11:38:07.264970 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.264973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.265030 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.265128 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.266106 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.266909 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.268232 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.268580 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.268714 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.270981 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.270984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.270986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.271014 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.273146 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.273678 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.273683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.273684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.273791 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.274098 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.274103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.274107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.274348 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.274451 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.274814 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.274954 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:38:07.271823 13881 master_runner.cc:186] Error getting master registration for 127.0.98.62:41613: OK, instance_id { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" instance_seqno: 1790422686716731 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:38:07.277211 13881 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:38:07.280725 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:07.281001 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:38:07.282526 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.264008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.0.98.60:40481)
0926 11:38:07.264142 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.264147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.271890 (+ 7743us) server_negotiation.cc:407] Connection header received
0926 11:38:07.272076 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.272081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.272135 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.272225 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.273122 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.273862 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.274997 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.275316 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.275418 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.278473 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.278477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.278480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.278511 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.280484 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.281002 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.281006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.281008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.281098 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.281399 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.281403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.281405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.281672 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.281811 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.282229 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.282360 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 11:38:07.285323 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.264511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.0.98.61:40039)
0926 11:38:07.264619 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.264623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.272294 (+ 7671us) server_negotiation.cc:407] Connection header received
0926 11:38:07.275177 (+ 2883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.275181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.275237 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.275291 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.276358 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.277192 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.278056 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.278382 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.278500 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.280282 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.280289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.280293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.280329 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.282105 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.282615 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.282620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.282623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.282842 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.284221 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.284224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.284226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.284467 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.284587 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.285005 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.285153 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:38:07.305074 13977 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:07.305322 13977 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } }
I20260926 11:38:07.306479 13977 leader_election.cc:288] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f41f17b8f8241308f4ee88885ff8804 (127.0.98.62:41613), bdefbb5127c04944a8d5980dd95dcacd (127.0.98.61:40039)
I20260926 11:38:07.307755 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdefbb5127c04944a8d5980dd95dcacd" is_pre_election: true
I20260926 11:38:07.308149 13845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0d0d1b8874e4f6fa936101c863a7b5d in term 0.
I20260926 11:38:07.308717 13900 leader_election.cc:302] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdefbb5127c04944a8d5980dd95dcacd, c0d0d1b8874e4f6fa936101c863a7b5d; no voters:
I20260926 11:38:07.309089 13977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:38:07.309203 13977 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:38:07.309248 13977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:07.310702 13977 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } }
I20260926 11:38:07.311156 13977 leader_election.cc:288] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [CANDIDATE]: Term 1 election: Requested vote from peers 0f41f17b8f8241308f4ee88885ff8804 (127.0.98.62:41613), bdefbb5127c04944a8d5980dd95dcacd (127.0.98.61:40039)
I20260926 11:38:07.312361 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdefbb5127c04944a8d5980dd95dcacd"
I20260926 11:38:07.312523 13845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:07.314050 13845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0d0d1b8874e4f6fa936101c863a7b5d in term 1.
I20260926 11:38:07.314518 13900 leader_election.cc:302] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdefbb5127c04944a8d5980dd95dcacd, c0d0d1b8874e4f6fa936101c863a7b5d; no voters:
I20260926 11:38:07.314817 13977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:38:07.315052 13977 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [term 1 LEADER]: Becoming Leader. State: Replica: c0d0d1b8874e4f6fa936101c863a7b5d, State: Running, Role: LEADER
I20260926 11:38:07.315476 13977 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } }
I20260926 11:38:07.319196 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d0d1b8874e4f6fa936101c863a7b5d. Latest consensus state: current_term: 1 leader_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:07.319676 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:07.320653 13992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:38:07.322990 13992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:38:07.332609 13845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [term 1 FOLLOWER]: Refusing update from remote peer c0d0d1b8874e4f6fa936101c863a7b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:07.333585 13990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:07.338222 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d0d1b8874e4f6fa936101c863a7b5d. Latest consensus state: current_term: 1 leader_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:07.338375 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:38:07.344810 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:07.345019 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:07.345206 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:07.345326 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:38:07.345446 13993 mvcc.cc:205] Tried to move back new op lower bound from 7333571327306452992 to 7333571327249833984. Current Snapshot: MvccSnapshot[applied={T|T < 7333571327306452992}]
I20260926 11:38:07.346122 13994 mvcc.cc:205] Tried to move back new op lower bound from 7333571327306452992 to 7333571327249833984. Current Snapshot: MvccSnapshot[applied={T|T < 7333571327306452992}]
I20260926 11:38:07.348762 13992 catalog_manager.cc:1347] Generated new cluster ID: f5e16ffb27cc43eebc811c1228db59e0
I20260926 11:38:07.348845 13992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:38:07.363548 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } attempt: 4
I20260926 11:38:07.373349 13992 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:38:07.374696 13992 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 11:38:07 dist-test-slave-4pw5 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:38:07.385802 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.364194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40481 (local address 127.0.0.1:54782)
0926 11:38:07.364508 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:07.364523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:07.364622 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:07.365387 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:07.365391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:07.365618 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:07.365728 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.365735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.369322 (+ 3587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.369326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.370128 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:07.370139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.372392 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.372396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:07.372491 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.374124 (+ 1633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:07.374140 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:07.375867 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:07.378387 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.378390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.378393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.378656 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.379168 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:07.379171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:07.379173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:07.379300 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:07.379995 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:07.379998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:07.380121 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 11:38:07.380206 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:38:07.380370 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260926 11:38:07.386480 13992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d: Generated new TSK 0
I20260926 11:38:07.387426 13992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:38:07.388244 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:07.364778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.98.60:40481)
0926 11:38:07.364903 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:07.364908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:07.364922 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:38:07.364988 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:07.364991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:07.365038 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:07.365119 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:07.365871 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.366664 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.371296 (+ 4632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:07.371744 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:07.371846 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:07.376074 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:07.376078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:07.376082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:07.376109 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:07.378108 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.378951 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.378955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.378957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.379045 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:07.379439 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:07.379442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:07.379444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:07.379753 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:07.383194 (+ 3441us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:07.383285 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 11:38:07.385338 (+ 2053us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:38:07.391942 13799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804: Bootstrap starting.
I20260926 11:38:07.404276 13799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:07.405525 13799 log.cc:824] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:07.409607 13799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804: No bootstrap required, opened a new log
I20260926 11:38:07.414393 13799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } }
I20260926 11:38:07.414680 13799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:07.414734 13799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f41f17b8f8241308f4ee88885ff8804, State: Initialized, Role: FOLLOWER
I20260926 11:38:07.415303 13799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } }
I20260926 11:38:07.417162 13799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:38:07.417491 13771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:07.419243 13771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [term 1 FOLLOWER]: Refusing update from remote peer c0d0d1b8874e4f6fa936101c863a7b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:07.420150 13990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:07.422705 13773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f41f17b8f8241308f4ee88885ff8804" is_pre_election: true
I20260926 11:38:07.423039 13773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0d0d1b8874e4f6fa936101c863a7b5d in term 1.
I20260926 11:38:07.423386 13772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f41f17b8f8241308f4ee88885ff8804"
W20260926 11:38:07.424015 14015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:38:07.424103 14015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:38:07.426786 13772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0d0d1b8874e4f6fa936101c863a7b5d in term 1.
I20260926 11:38:07.420534 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:07.440412 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:38:07.440641 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:07.440834 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:38:07.444253 14017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d0d1b8874e4f6fa936101c863a7b5d. Latest consensus state: current_term: 1 leader_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:07.444456 14017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:38:07.452512 14018 mvcc.cc:205] Tried to move back new op lower bound from 7333571327520620544 to 7333571327249833984. Current Snapshot: MvccSnapshot[applied={T|T < 7333571327306452992}]
I20260926 11:38:08.024588 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f41f17b8f8241308f4ee88885ff8804" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } peers { permanent_uuid: "bdefbb5127c04944a8d5980dd95dcacd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40039 } } peers { permanent_uuid: "c0d0d1b8874e4f6fa936101c863a7b5d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40481 } } }
I20260926 11:38:08.024864 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d0d1b8874e4f6fa936101c863a7b5d [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:08.261365 13967 catalog_manager.cc:1259] Loaded cluster ID: f5e16ffb27cc43eebc811c1228db59e0
I20260926 11:38:08.261488 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd: loading cluster ID for follower catalog manager: success
I20260926 11:38:08.263818 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd: acquiring CA information for follower catalog manager: success
I20260926 11:38:08.265442 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bdefbb5127c04944a8d5980dd95dcacd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:38:08.425886 14015 catalog_manager.cc:1259] Loaded cluster ID: f5e16ffb27cc43eebc811c1228db59e0
I20260926 11:38:08.426020 14015 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804: loading cluster ID for follower catalog manager: success
I20260926 11:38:08.428002 14015 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804: acquiring CA information for follower catalog manager: success
I20260926 11:38:08.429407 14015 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f41f17b8f8241308f4ee88885ff8804: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:38:08.485376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 13731
I20260926 11:38:08.493960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 13806
I20260926 11:38:08.501689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 13881
2026-09-26T11:38:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3268 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:38:08 dist-test-slave-4pw5 krb5kdc[14029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:38:08 dist-test-slave-4pw5 krb5kdc[14029](info): set up 2 sockets
Sep 26 11:38:08 dist-test-slave-4pw5 krb5kdc[14029](info): commencing operation
krb5kdc: starting...
W20260926 11:38:09.588064 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:38:09 dist-test-slave-4pw5 krb5kdc[14029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:38:09.728914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:09.848994 14042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:09.849357 14042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:09.849443 14042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:09.853698 14042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:09.853826 14042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:09.853859 14042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:09.853899 14042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:09.853927 14042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:09.859786 14042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:09.861487 14042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:09.862799 14042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:09.872740 14048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:09.872797 14050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:09.872823 14047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:09.873049 14042 server_base.cc:1034] running on GCE node
I20260926 11:38:09.873699 14042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:09.874924 14042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:09.876101 14042 hybrid_clock.cc:648] HybridClock initialized: now 1790422689876035 us; error 81 us; skew 500 ppm
Sep 26 11:38:09 dist-test-slave-4pw5 krb5kdc[14029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:09.879136 14042 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:38:09.880224 14042 webserver.cc:460] Webserver started at http://127.0.98.62:36755/ using document root <none> and password file <none>
I20260926 11:38:09.880834 14042 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:09.880903 14042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:09.881076 14042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:09.883183 14042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "a072c2b8e4444c139b2408774a78d96f"
format_stamp: "Formatted at 2026-09-26 11:38:09 on dist-test-slave-4pw5"
server_key: "030ae7a7b2d60a6e609fa3a41a84c1a9"
server_key_iv: "bab975751a2b4e2f0da1639e7e283e80"
server_key_version: "encryptionkey@0"
I20260926 11:38:09.883852 14042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "a072c2b8e4444c139b2408774a78d96f"
format_stamp: "Formatted at 2026-09-26 11:38:09 on dist-test-slave-4pw5"
server_key: "030ae7a7b2d60a6e609fa3a41a84c1a9"
server_key_iv: "bab975751a2b4e2f0da1639e7e283e80"
server_key_version: "encryptionkey@0"
I20260926 11:38:09.887887 14042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:38:09.890503 14057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:09.891927 14042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:09.892095 14042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "a072c2b8e4444c139b2408774a78d96f"
format_stamp: "Formatted at 2026-09-26 11:38:09 on dist-test-slave-4pw5"
server_key: "030ae7a7b2d60a6e609fa3a41a84c1a9"
server_key_iv: "bab975751a2b4e2f0da1639e7e283e80"
server_key_version: "encryptionkey@0"
I20260926 11:38:09.892235 14042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:09.917548 14042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:09.921456 14042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:09.921708 14042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:09.929400 14042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34095
I20260926 11:38:09.929425 14109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34095 every 8 connection(s)
I20260926 11:38:09.930416 14042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:38:09.933480 14110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:09.936666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 14042
I20260926 11:38:09.936807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:38:09.937111 392 external_mini_cluster.cc:1428] Setting key 2920cd8d98fc20444ab5898e30aeeb83
I20260926 11:38:09.940335 14110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f: Bootstrap starting.
I20260926 11:38:09.942865 14110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:09.943948 14110 log.cc:824] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:09.946597 14110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f: No bootstrap required, opened a new log
Sep 26 11:38:09 dist-test-slave-4pw5 krb5kdc[14029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:09.951406 14110 raft_consensus.cc:348] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a072c2b8e4444c139b2408774a78d96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260926 11:38:09.951797 14110 raft_consensus.cc:374] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:09.951860 14110 raft_consensus.cc:729] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a072c2b8e4444c139b2408774a78d96f, State: Initialized, Role: FOLLOWER
I20260926 11:38:09.952526 14110 consensus_queue.cc:260] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a072c2b8e4444c139b2408774a78d96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260926 11:38:09.952708 14110 raft_consensus.cc:388] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:38:09.952818 14110 raft_consensus.cc:482] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:38:09.952955 14110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:09.954291 14110 raft_consensus.cc:504] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a072c2b8e4444c139b2408774a78d96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260926 11:38:09.954711 14110 leader_election.cc:302] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a072c2b8e4444c139b2408774a78d96f; no voters:
I20260926 11:38:09.955096 14110 leader_election.cc:288] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:38:09.955233 14115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:38:09.955485 14115 raft_consensus.cc:686] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [term 1 LEADER]: Becoming Leader. State: Replica: a072c2b8e4444c139b2408774a78d96f, State: Running, Role: LEADER
I20260926 11:38:09.955483 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:09.938700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.0.98.62:34095)
0926 11:38:09.939231 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:09.939247 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:09.939300 (+ 53us) server_negotiation.cc:407] Connection header received
0926 11:38:09.940100 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:09.940125 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:09.940551 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:09.940932 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:09.942035 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:09.943130 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:09.944426 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:09.945139 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:09.945323 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:09.948797 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:09.948813 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:09.948839 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:09.948882 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:09.951763 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:09.952438 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:09.952444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:09.952451 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:09.952614 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:09.952975 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:09.952980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:09.952982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:09.953556 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:09.953777 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:09.954054 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 11:38:09.954405 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":156,"threads_started":1}
I20260926 11:38:09.956023 14115 consensus_queue.cc:237] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a072c2b8e4444c139b2408774a78d96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } }
I20260926 11:38:09.956486 14110 sys_catalog.cc:564] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:38:09.957896 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a072c2b8e4444c139b2408774a78d96f. Latest consensus state: current_term: 1 leader_uuid: "a072c2b8e4444c139b2408774a78d96f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a072c2b8e4444c139b2408774a78d96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } } }
I20260926 11:38:09.958048 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:09.958376 14116 sys_catalog.cc:455] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a072c2b8e4444c139b2408774a78d96f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a072c2b8e4444c139b2408774a78d96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34095 } } }
I20260926 11:38:09.958474 14116 sys_catalog.cc:458] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:09.958477 14126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:38:09.959151 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:38:09.962194 14126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:38:09.968403 14126 catalog_manager.cc:1347] Generated new cluster ID: 13c32bf699b440d7aa66f3b9826d0afa
I20260926 11:38:09.968502 14126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:38:09.977231 14126 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:38:09.978258 14126 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:38:09.985821 14126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a072c2b8e4444c139b2408774a78d96f: Generated new TSK 0
I20260926 11:38:09.986581 14126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:38:10.016101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 14042
2026-09-26T11:38:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1511 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:38:10 dist-test-slave-4pw5 krb5kdc[14137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:38:10 dist-test-slave-4pw5 krb5kdc[14137](info): set up 2 sockets
Sep 26 11:38:10 dist-test-slave-4pw5 krb5kdc[14137](info): commencing operation
krb5kdc: starting...
W20260926 11:38:11.088439 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:38:11 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:38:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:38:11.241077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:11.360422 14150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:11.360765 14150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:11.360808 14150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:11.365145 14150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:11.365229 14150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:11.365250 14150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:11.365267 14150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:11.365283 14150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:11.371033 14150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:11.372285 14150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:11.373381 14150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:11.382674 14155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:11.382696 14158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:11.382680 14156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:11.382746 14150 server_base.cc:1034] running on GCE node
I20260926 11:38:11.383922 14150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:11.385294 14150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:11.386543 14150 hybrid_clock.cc:648] HybridClock initialized: now 1790422691386523 us; error 39 us; skew 500 ppm
Sep 26 11:38:11 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:11.389962 14150 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:38:11.391155 14150 webserver.cc:460] Webserver started at http://127.0.98.62:32769/ using document root <none> and password file <none>
I20260926 11:38:11.391901 14150 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:11.391976 14150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:11.392233 14150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:11.394629 14150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "a73eedb1dbc52ba0cc5344736ab89f31"
server_key_iv: "5a589e33382780ac9ae64fc38288e1aa"
server_key_version: "encryptionkey@0"
I20260926 11:38:11.395314 14150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "a73eedb1dbc52ba0cc5344736ab89f31"
server_key_iv: "5a589e33382780ac9ae64fc38288e1aa"
server_key_version: "encryptionkey@0"
I20260926 11:38:11.400280 14150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 11:38:11.403100 14165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:11.404717 14150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:11.404906 14150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "a73eedb1dbc52ba0cc5344736ab89f31"
server_key_iv: "5a589e33382780ac9ae64fc38288e1aa"
server_key_version: "encryptionkey@0"
I20260926 11:38:11.405066 14150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:11.422699 14150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:11.426388 14150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:11.426626 14150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:11.434473 14150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40241
I20260926 11:38:11.434886 14217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40241 every 8 connection(s)
I20260926 11:38:11.435539 14150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:38:11.438869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 14150
I20260926 11:38:11.439018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:38:11.439092 14218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:11.439361 392 external_mini_cluster.cc:1428] Setting key 8d14c79bf1ef018ae6796e594092b51b
I20260926 11:38:11.447250 14218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Bootstrap starting.
Sep 26 11:38:11 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:11.450098 14218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:11.450991 14218 log.cc:824] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:11.453397 14218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: No bootstrap required, opened a new log
I20260926 11:38:11.455345 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:11.440997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.0.98.62:40241)
0926 11:38:11.441511 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:11.441527 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:11.441564 (+ 37us) server_negotiation.cc:407] Connection header received
0926 11:38:11.442482 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:11.442507 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:11.442906 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:11.443551 (+ 645us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:11.444507 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:11.445556 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:11.446469 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:11.447006 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:11.447193 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:11.449922 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:11.449929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:11.449945 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:11.449986 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:11.452222 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:11.452772 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:11.452777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:11.452783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:11.452901 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:11.453215 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:11.453219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:11.453221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:11.453722 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:11.453933 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:11.454246 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 11:38:11.454564 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":175,"threads_started":1}
I20260926 11:38:11.458302 14218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:11.458631 14218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:11.458729 14218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1888e607c9fb48bf98ac8f01bb8d7d0b, State: Initialized, Role: FOLLOWER
I20260926 11:38:11.459230 14218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:11.459422 14218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:38:11.459528 14218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:38:11.459699 14218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:11.461001 14218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:11.461465 14218 leader_election.cc:302] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1888e607c9fb48bf98ac8f01bb8d7d0b; no voters:
I20260926 11:38:11.461910 14218 leader_election.cc:288] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:38:11.462030 14223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:38:11.462297 14223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 1 LEADER]: Becoming Leader. State: Replica: 1888e607c9fb48bf98ac8f01bb8d7d0b, State: Running, Role: LEADER
I20260926 11:38:11.462683 14223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:11.463449 14218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:38:11.463918 14223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1888e607c9fb48bf98ac8f01bb8d7d0b. Latest consensus state: current_term: 1 leader_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } } }
I20260926 11:38:11.464095 14223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:11.464535 14227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:38:11.464445 14224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } } }
I20260926 11:38:11.464783 14224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:11.468000 14227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:38:11.474992 14227 catalog_manager.cc:1347] Generated new cluster ID: a2b017d2ba50468d90ab90e53a183d09
I20260926 11:38:11.475142 14227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:38:11.488391 14227 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:38:11.489918 14227 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:38:11.502375 14227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Generated new TSK 0
I20260926 11:38:11.503312 14227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:38:11.539929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40241
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:11.675417 14244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:11.675770 14244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:11.675818 14244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:11.680124 14244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:11.680202 14244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:11.680264 14244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:38:11.685956 14244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:11.687505 14244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:11.688695 14244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:11.699908 14250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:11.699908 14249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:11.699987 14252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:11.700001 14244 server_base.cc:1034] running on GCE node
I20260926 11:38:11.700681 14244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:11.701375 14244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:11.702569 14244 hybrid_clock.cc:648] HybridClock initialized: now 1790422691702542 us; error 43 us; skew 500 ppm
Sep 26 11:38:11 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:11.705590 14244 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:38:11.706584 14244 webserver.cc:460] Webserver started at http://127.0.98.1:37933/ using document root <none> and password file <none>
I20260926 11:38:11.707185 14244 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:11.707283 14244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:11.707536 14244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:11.709895 14244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "dd0f2c6e00c24e52b7b1078dc836a17a"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "799c95ce2842d3bca736a28598a2b38b"
server_key_iv: "7e5d9f7264b518baaaab30bf6ed001e6"
server_key_version: "encryptionkey@0"
I20260926 11:38:11.710552 14244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dd0f2c6e00c24e52b7b1078dc836a17a"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "799c95ce2842d3bca736a28598a2b38b"
server_key_iv: "7e5d9f7264b518baaaab30bf6ed001e6"
server_key_version: "encryptionkey@0"
I20260926 11:38:11.714665 14244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 11:38:11.717567 14259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:11.718953 14244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:38:11.719115 14244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "dd0f2c6e00c24e52b7b1078dc836a17a"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "799c95ce2842d3bca736a28598a2b38b"
server_key_iv: "7e5d9f7264b518baaaab30bf6ed001e6"
server_key_version: "encryptionkey@0"
I20260926 11:38:11.719249 14244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:11.733799 14244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:11.737639 14244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:11.737866 14244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:11.738523 14244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:11.739882 14244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:11.740007 14244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:11.740109 14244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:11.740160 14244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:11.752085 14244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34453
I20260926 11:38:11.752102 14372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34453 every 8 connection(s)
I20260926 11:38:11.753108 14244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:38:11.758623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 14244
I20260926 11:38:11.758736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:38:11.759070 392 external_mini_cluster.cc:1428] Setting key 53b6bfe40268f9968d1c88afb28899a1
Sep 26 11:38:11 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:11.770399 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:11.755067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38837 (local address 127.0.98.62:40241)
0926 11:38:11.755222 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:11.755226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:11.756204 (+ 978us) server_negotiation.cc:407] Connection header received
0926 11:38:11.757589 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:11.757592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:11.757648 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:11.757727 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:11.759478 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:11.760262 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:11.761674 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:11.761957 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:11.762102 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:11.765588 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:11.765597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:11.765605 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:11.765650 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:11.767635 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:11.768427 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:11.768434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:11.768437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:11.768550 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:11.768974 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:11.768978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:11.768980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:11.769277 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:11.769421 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:11.769983 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 11:38:11.770150 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:38:11.771270 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:11.755375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:38837)
0926 11:38:11.756009 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:11.756057 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:11.757339 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:11.757986 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:11.757995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:11.758473 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:11.759285 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:11.759305 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:11.760441 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:11.760445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:11.761515 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:11.761526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:11.762155 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:11.762159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:11.762360 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:11.763182 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:11.763202 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:11.765348 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:11.767798 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:11.767806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:11.767826 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:11.768262 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:11.768669 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:11.768671 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:11.768672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:11.768864 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:11.769420 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:11.769425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:11.769708 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 11:38:11.769876 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 11:38:11.770176 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":220,"threads_started":1}
I20260926 11:38:11.772789 14373 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40241
I20260926 11:38:11.773108 14373 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:11.773734 14373 heartbeater.cc:511] Master 127.0.98.62:40241 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:38:11.775880 14182 ts_manager.cc:194] Registered new tserver with Master: dd0f2c6e00c24e52b7b1078dc836a17a (127.0.98.1:34453)
I20260926 11:38:11.777211 14182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:38:11.794886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40241
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:11.912636 14378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:11.912937 14378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:11.912987 14378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:11.917245 14378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:11.917325 14378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:11.917419 14378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:38:11.923240 14378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:11.924531 14378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:11.925741 14378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:11.936326 14384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:11.936380 14378 server_base.cc:1034] running on GCE node
W20260926 11:38:11.936465 14383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:11.936307 14386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:11.936946 14378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:11.937628 14378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:11.938840 14378 hybrid_clock.cc:648] HybridClock initialized: now 1790422691938812 us; error 57 us; skew 500 ppm
Sep 26 11:38:11 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:11.942817 14378 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:38:11.943930 14378 webserver.cc:460] Webserver started at http://127.0.98.2:38211/ using document root <none> and password file <none>
I20260926 11:38:11.944509 14378 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:11.944571 14378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:11.944743 14378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:11.946895 14378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "973fe1bffa2f428a936f2b9215650bb1"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "2266b4ab3cd1d86d76de9f3749b7b390"
server_key_iv: "ef301027c613473ce3905b62118407a2"
server_key_version: "encryptionkey@0"
I20260926 11:38:11.947431 14378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "973fe1bffa2f428a936f2b9215650bb1"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "2266b4ab3cd1d86d76de9f3749b7b390"
server_key_iv: "ef301027c613473ce3905b62118407a2"
server_key_version: "encryptionkey@0"
I20260926 11:38:11.951421 14378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 11:38:11.954244 14393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:11.955722 14378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:11.955847 14378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "973fe1bffa2f428a936f2b9215650bb1"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "2266b4ab3cd1d86d76de9f3749b7b390"
server_key_iv: "ef301027c613473ce3905b62118407a2"
server_key_version: "encryptionkey@0"
I20260926 11:38:11.955997 14378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:11.988358 14378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:11.992012 14378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:11.992225 14378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:11.992865 14378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:11.993714 14378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:11.993760 14378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:11.993831 14378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:11.993871 14378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:12.003578 14378 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45307
I20260926 11:38:12.003612 14506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45307 every 8 connection(s)
I20260926 11:38:12.004660 14378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:38:12.012470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 14378
I20260926 11:38:12.012573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:38:12.012884 392 external_mini_cluster.cc:1428] Setting key 084c9e8116fbf2475cf4b51d639d99ba
Sep 26 11:38:12 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:12.020716 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.006683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48791 (local address 127.0.98.62:40241)
0926 11:38:12.006826 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.006830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.007925 (+ 1095us) server_negotiation.cc:407] Connection header received
0926 11:38:12.009303 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.009306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.009364 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.009445 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:12.010735 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.011406 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.012980 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.013234 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.013379 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.016545 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:12.016549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:12.016551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:12.016578 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:12.018353 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:12.019000 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:12.019009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:12.019011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:12.019093 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:12.019475 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:12.019479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:12.019481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:12.019755 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:12.019862 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.020414 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.020550 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:38:12.021624 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.007111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:48791)
0926 11:38:12.007766 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:12.007811 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:12.009096 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:12.009621 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:12.009627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:12.010066 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:12.010569 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.010582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.011768 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.011771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.012819 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.012839 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.013439 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.013444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.013637 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.014322 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:12.014343 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:12.016282 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:12.018516 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:12.018523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:12.018544 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:12.018876 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:12.019205 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:12.019208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:12.019210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:12.019366 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:12.019873 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:12.019883 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:12.020161 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 11:38:12.020360 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 11:38:12.020722 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:38:12.022900 14507 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40241
I20260926 11:38:12.023204 14507 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:12.023886 14507 heartbeater.cc:511] Master 127.0.98.62:40241 requested a full tablet report, sending...
I20260926 11:38:12.025101 14182 ts_manager.cc:194] Registered new tserver with Master: 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:38:12.025928 14182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:38:12.044788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40241
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:12.164834 14512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:12.165153 14512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:12.165200 14512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:12.169574 14512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:12.169657 14512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:12.169723 14512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:38:12.175637 14512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:12.176838 14512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:12.177911 14512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:12.189000 14517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:12.189000 14518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:12.189065 14520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:12.189136 14512 server_base.cc:1034] running on GCE node
I20260926 11:38:12.190011 14512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:12.190729 14512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:12.194039 14512 hybrid_clock.cc:648] HybridClock initialized: now 1790422692194023 us; error 82 us; skew 500 ppm
Sep 26 11:38:12 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:12.197520 14512 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:38:12.198607 14512 webserver.cc:460] Webserver started at http://127.0.98.3:42541/ using document root <none> and password file <none>
I20260926 11:38:12.199252 14512 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:12.199358 14512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:12.199671 14512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:12.201989 14512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b6557ae601448db8938555056bb09dd"
format_stamp: "Formatted at 2026-09-26 11:38:12 on dist-test-slave-4pw5"
server_key: "ea6ab89e8b72633a78861989fae8d8b9"
server_key_iv: "aa8601a81413f9b00c6dcb2732e2e558"
server_key_version: "encryptionkey@0"
I20260926 11:38:12.202688 14512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b6557ae601448db8938555056bb09dd"
format_stamp: "Formatted at 2026-09-26 11:38:12 on dist-test-slave-4pw5"
server_key: "ea6ab89e8b72633a78861989fae8d8b9"
server_key_iv: "aa8601a81413f9b00c6dcb2732e2e558"
server_key_version: "encryptionkey@0"
I20260926 11:38:12.207492 14512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 11:38:12.210855 14527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:12.212541 14512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 11:38:12.212709 14512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "3b6557ae601448db8938555056bb09dd"
format_stamp: "Formatted at 2026-09-26 11:38:12 on dist-test-slave-4pw5"
server_key: "ea6ab89e8b72633a78861989fae8d8b9"
server_key_iv: "aa8601a81413f9b00c6dcb2732e2e558"
server_key_version: "encryptionkey@0"
I20260926 11:38:12.212852 14512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:12.239482 14512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:12.243191 14512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:12.243445 14512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:12.244177 14512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:12.245111 14512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:12.245191 14512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:12.245260 14512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:12.245308 14512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:12.255177 14512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41639
I20260926 11:38:12.255193 14640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41639 every 8 connection(s)
I20260926 11:38:12.256294 14512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:38:12.262576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 14512
I20260926 11:38:12.262775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:38:12.263092 392 external_mini_cluster.cc:1428] Setting key c04092b4a158491052ac33a3d0c2f293
Sep 26 11:38:12 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:12.272032 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.258452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42733 (local address 127.0.98.62:40241)
0926 11:38:12.258664 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.258669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.260022 (+ 1353us) server_negotiation.cc:407] Connection header received
0926 11:38:12.261148 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.261151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.261213 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.261298 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:12.262561 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.263331 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.265019 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.265268 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.265382 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.268177 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:12.268181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:12.268184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:12.268222 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:12.269746 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:12.270345 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:12.270349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:12.270351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:12.270437 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:12.270811 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:12.270815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:12.270817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:12.271021 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:12.271133 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.271713 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.271858 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 11:38:12.273015 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.258813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:42733)
0926 11:38:12.259835 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:12.259882 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:12.260900 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:12.261495 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:12.261502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:12.261875 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:12.262398 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.262413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.263452 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.263457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.264894 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.264904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.265436 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.265440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.265685 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.266394 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:12.266411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:12.267979 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:12.269896 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:12.269904 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:12.269924 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:12.270223 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:12.270538 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:12.270542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:12.270544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:12.270705 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:12.271129 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:12.271138 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:12.271425 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 11:38:12.271657 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 11:38:12.272066 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":707,"threads_started":1}
I20260926 11:38:12.274492 14641 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40241
I20260926 11:38:12.274763 14641 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:12.275318 14641 heartbeater.cc:511] Master 127.0.98.62:40241 requested a full tablet report, sending...
I20260926 11:38:12.276643 14182 ts_manager.cc:194] Registered new tserver with Master: 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639)
I20260926 11:38:12.277443 14182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42733
I20260926 11:38:12.277945 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:38:12.288865 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.280306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.0.98.62:40241)
0926 11:38:12.280453 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.280458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.280615 (+ 157us) server_negotiation.cc:407] Connection header received
0926 11:38:12.280670 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.280672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.280715 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.280791 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:12.281587 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.282122 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.282993 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.283216 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.283300 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.285106 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:12.285109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:12.285111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:12.285137 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:12.286735 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:12.287197 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:12.287200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:12.287201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:12.287277 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:12.287704 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:12.287707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:12.287708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:12.288013 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:12.288140 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.288518 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.288656 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:38:12.292552 14182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:38:12.296200 14182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:38:12.312711 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.306057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.0.98.1:34453)
0926 11:38:12.306466 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.306473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.306925 (+ 452us) server_negotiation.cc:407] Connection header received
0926 11:38:12.307012 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.307017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.307237 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.307382 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:12.308070 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.309281 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.311102 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.311807 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.311920 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.312087 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.312411 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.312534 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":91,"threads_started":1}
I20260926 11:38:12.312704 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.305780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34453 (local address 127.0.0.1:47314)
0926 11:38:12.306552 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:12.306610 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:12.306773 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:12.307604 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:12.307613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:12.307641 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:12.307944 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.307950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.309451 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.309454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.310974 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.310983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.311939 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.311943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.312175 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.312234 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 11:38:12.312359 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 11:38:12.312522 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":84,"spinlock_wait_cycles":8832,"thread_start_us":120,"threads_started":1}
I20260926 11:38:12.312841 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.306631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45307 (local address 127.0.0.1:38718)
0926 11:38:12.307014 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:12.307026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:12.307121 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:12.308018 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:12.308021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:12.308039 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:12.308151 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.308160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.309653 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.309657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.311071 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.311081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.312451 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.312455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.312535 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.312555 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:38:12.312608 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 11:38:12.312668 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":149,"thread_start_us":102,"threads_started":1}
I20260926 11:38:12.313153 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.305780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41639 (local address 127.0.0.1:58796)
0926 11:38:12.306588 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:12.306625 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:12.306772 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:12.307679 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:12.307682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:12.307694 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:12.307849 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.307857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.309457 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.309461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.311677 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.311687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.312733 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.312739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.312831 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.312857 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:38:12.312911 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 11:38:12.312996 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":1014,"spinlock_wait_cycles":9984,"thread_start_us":102,"threads_started":1}
I20260926 11:38:12.313442 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.306798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.0.98.2:45307)
0926 11:38:12.307467 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.307486 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.307514 (+ 28us) server_negotiation.cc:407] Connection header received
0926 11:38:12.307630 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.307637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.307876 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.308040 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:12.308295 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.309472 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.311211 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.312259 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.312961 (+ 702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.313128 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.313183 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.313286 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":117,"threads_started":1}
I20260926 11:38:12.314482 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.305993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.0.98.3:41639)
0926 11:38:12.307163 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.307176 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.307252 (+ 76us) server_negotiation.cc:407] Connection header received
0926 11:38:12.307328 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.307334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.307501 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.307885 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:12.308078 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.309281 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.311807 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.312590 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.314023 (+ 1433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.314176 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.314224 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.314317 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":157,"threads_started":1}
I20260926 11:38:12.316325 14307 tablet_service.cc:1467] Processing CreateTablet for tablet da369d47d95a4bd4b77185b2fe7b766f (DEFAULT_TABLE table=test-table [id=9185cffdece545f3a7ac6f8684fe520e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:12.316329 14441 tablet_service.cc:1467] Processing CreateTablet for tablet da369d47d95a4bd4b77185b2fe7b766f (DEFAULT_TABLE table=test-table [id=9185cffdece545f3a7ac6f8684fe520e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:12.317144 14575 tablet_service.cc:1467] Processing CreateTablet for tablet da369d47d95a4bd4b77185b2fe7b766f (DEFAULT_TABLE table=test-table [id=9185cffdece545f3a7ac6f8684fe520e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:12.317551 14307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da369d47d95a4bd4b77185b2fe7b766f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:12.317551 14441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da369d47d95a4bd4b77185b2fe7b766f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:12.318167 14575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da369d47d95a4bd4b77185b2fe7b766f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:12.325992 14658 tablet_bootstrap.cc:492] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: Bootstrap starting.
I20260926 11:38:12.327945 14660 tablet_bootstrap.cc:492] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: Bootstrap starting.
I20260926 11:38:12.328209 14658 tablet_bootstrap.cc:654] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:12.328505 14659 tablet_bootstrap.cc:492] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Bootstrap starting.
I20260926 11:38:12.329403 14658 log.cc:824] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:12.330828 14660 tablet_bootstrap.cc:654] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:12.330943 14659 tablet_bootstrap.cc:654] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:12.331784 14660 log.cc:824] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:12.331785 14659 log.cc:824] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:12.331833 14658 tablet_bootstrap.cc:492] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: No bootstrap required, opened a new log
I20260926 11:38:12.332048 14658 ts_tablet_manager.cc:1397] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:38:12.333695 14659 tablet_bootstrap.cc:492] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: No bootstrap required, opened a new log
I20260926 11:38:12.333832 14660 tablet_bootstrap.cc:492] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: No bootstrap required, opened a new log
I20260926 11:38:12.333954 14659 ts_tablet_manager.cc:1397] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:38:12.334048 14660 ts_tablet_manager.cc:1397] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:38:12.336916 14658 raft_consensus.cc:348] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:12.337338 14658 raft_consensus.cc:374] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:12.337399 14658 raft_consensus.cc:729] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6557ae601448db8938555056bb09dd, State: Initialized, Role: FOLLOWER
I20260926 11:38:12.337611 14659 raft_consensus.cc:348] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:12.337893 14659 raft_consensus.cc:374] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:12.337952 14659 raft_consensus.cc:729] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0f2c6e00c24e52b7b1078dc836a17a, State: Initialized, Role: FOLLOWER
I20260926 11:38:12.338001 14658 consensus_queue.cc:260] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:12.338407 14659 consensus_queue.cc:260] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:12.338609 14660 raft_consensus.cc:348] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:12.338907 14660 raft_consensus.cc:374] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:12.338971 14660 raft_consensus.cc:729] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973fe1bffa2f428a936f2b9215650bb1, State: Initialized, Role: FOLLOWER
I20260926 11:38:12.339263 14658 ts_tablet_manager.cc:1428] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 11:38:12.339349 14373 heartbeater.cc:503] Master 127.0.98.62:40241 was elected leader, sending a full tablet report...
I20260926 11:38:12.339426 14659 ts_tablet_manager.cc:1428] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 11:38:12.339706 14660 consensus_queue.cc:260] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:12.339959 14641 heartbeater.cc:503] Master 127.0.98.62:40241 was elected leader, sending a full tablet report...
I20260926 11:38:12.340659 14507 heartbeater.cc:503] Master 127.0.98.62:40241 was elected leader, sending a full tablet report...
I20260926 11:38:12.340834 14660 ts_tablet_manager.cc:1428] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260926 11:38:12.430002 14665 raft_consensus.cc:482] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:12.430285 14665 raft_consensus.cc:504] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:12.431785 14665 leader_election.cc:288] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639), 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307)
I20260926 11:38:12.436344 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.432251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35587 (local address 127.0.98.3:41639)
0926 11:38:12.432384 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.432388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.432637 (+ 249us) server_negotiation.cc:407] Connection header received
0926 11:38:12.432793 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.432796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.432872 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.432959 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:12.433293 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.433866 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.435107 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.435722 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.435826 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.435895 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.436134 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.436215 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 11:38:12.436363 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.432029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41639 (local address 127.0.98.1:35587)
0926 11:38:12.432553 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:12.432567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:12.432673 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:12.432992 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:12.432995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:12.433010 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:12.433162 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.433166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.433997 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.434000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.434971 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.434982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.435866 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.435870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.435984 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.436013 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 11:38:12.436099 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 11:38:12.436181 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":34,"thread_start_us":126,"threads_started":1}
I20260926 11:38:12.436754 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.432024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45307 (local address 127.0.98.1:55683)
0926 11:38:12.432471 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:12.432489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:12.432614 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:12.432907 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:12.432913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:12.432934 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:12.433151 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.433158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.433927 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.433935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.435247 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.435259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.436267 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.436271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.436439 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.436466 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 11:38:12.436540 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 11:38:12.436613 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":85,"thread_start_us":130,"threads_started":1}
I20260926 11:38:12.436748 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.432130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55683 (local address 127.0.98.2:45307)
0926 11:38:12.432258 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.432265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.432572 (+ 307us) server_negotiation.cc:407] Connection header received
0926 11:38:12.432733 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.432736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.432790 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.432859 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:12.433268 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.433797 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.435401 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.436033 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.436153 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.436207 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.436567 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.436639 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:38:12.437234 14595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da369d47d95a4bd4b77185b2fe7b766f" candidate_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6557ae601448db8938555056bb09dd" is_pre_election: true
I20260926 11:38:12.437595 14595 raft_consensus.cc:2454] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd0f2c6e00c24e52b7b1078dc836a17a in term 0.
I20260926 11:38:12.437690 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da369d47d95a4bd4b77185b2fe7b766f" candidate_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973fe1bffa2f428a936f2b9215650bb1" is_pre_election: true
I20260926 11:38:12.437954 14461 raft_consensus.cc:2454] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd0f2c6e00c24e52b7b1078dc836a17a in term 0.
I20260926 11:38:12.438140 14262 leader_election.cc:302] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6557ae601448db8938555056bb09dd, dd0f2c6e00c24e52b7b1078dc836a17a; no voters:
I20260926 11:38:12.438457 14665 raft_consensus.cc:2781] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:38:12.438529 14665 raft_consensus.cc:482] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:38:12.438558 14665 raft_consensus.cc:3037] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:12.439734 14665 raft_consensus.cc:504] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:12.440172 14665 leader_election.cc:288] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 1 election: Requested vote from peers 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639), 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307)
I20260926 11:38:12.440728 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da369d47d95a4bd4b77185b2fe7b766f" candidate_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973fe1bffa2f428a936f2b9215650bb1"
I20260926 11:38:12.440912 14461 raft_consensus.cc:3037] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:12.440920 14595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da369d47d95a4bd4b77185b2fe7b766f" candidate_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6557ae601448db8938555056bb09dd"
I20260926 11:38:12.441079 14595 raft_consensus.cc:3037] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:12.442263 14461 raft_consensus.cc:2454] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd0f2c6e00c24e52b7b1078dc836a17a in term 1.
I20260926 11:38:12.442468 14595 raft_consensus.cc:2454] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd0f2c6e00c24e52b7b1078dc836a17a in term 1.
I20260926 11:38:12.442672 14263 leader_election.cc:302] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 973fe1bffa2f428a936f2b9215650bb1, dd0f2c6e00c24e52b7b1078dc836a17a; no voters:
I20260926 11:38:12.442909 14665 raft_consensus.cc:2781] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:38:12.443186 14665 raft_consensus.cc:686] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 LEADER]: Becoming Leader. State: Replica: dd0f2c6e00c24e52b7b1078dc836a17a, State: Running, Role: LEADER
I20260926 11:38:12.443820 14665 consensus_queue.cc:237] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:12.448109 14182 catalog_manager.cc:5495] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a reported cstate change: term changed from 0 to 1, leader changed from <none> to dd0f2c6e00c24e52b7b1078dc836a17a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } health_report { overall_health: UNKNOWN } } }
I20260926 11:38:12.471794 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.467866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.0.98.1:34453)
0926 11:38:12.468031 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.468035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.468053 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:38:12.468142 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.468144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.468205 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.468320 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:38:12.468665 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.469193 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.470091 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.470313 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.470431 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.470741 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:38:12.471215 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:38:12.471312 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.471531 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.471674 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 11:38:12.478497 14595 raft_consensus.cc:1261] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [term 1 FOLLOWER]: Refusing update from remote peer dd0f2c6e00c24e52b7b1078dc836a17a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:12.478475 14461 raft_consensus.cc:1261] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [term 1 FOLLOWER]: Refusing update from remote peer dd0f2c6e00c24e52b7b1078dc836a17a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:12.479357 14669 consensus_queue.cc:1035] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:12.479650 14665 consensus_queue.cc:1035] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [LEADER]: Connected to new peer: Peer: permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:12.489208 14674 mvcc.cc:205] Tried to move back new op lower bound from 7333571348383547392 to 7333571348253409280. Current Snapshot: MvccSnapshot[applied={T|T < 7333571348383547392}]
I20260926 11:38:12.493865 14673 mvcc.cc:205] Tried to move back new op lower bound from 7333571348383547392 to 7333571348253409280. Current Snapshot: MvccSnapshot[applied={T|T < 7333571348383547392 or (T in {7333571348383547392})}]
I20260926 11:38:12.498974 14181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:12.499185 14181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:12.502142 14181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Sending DeleteTablet for 3 replicas of tablet da369d47d95a4bd4b77185b2fe7b766f
I20260926 11:38:12.503139 14575 tablet_service.cc:1514] Processing DeleteTablet for tablet da369d47d95a4bd4b77185b2fe7b766f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58796
I20260926 11:38:12.503235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 14244
I20260926 11:38:12.503388 14441 tablet_service.cc:1514] Processing DeleteTablet for tablet da369d47d95a4bd4b77185b2fe7b766f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38718
I20260926 11:38:12.504150 14683 tablet_replica.cc:330] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: stopping tablet replica
I20260926 11:38:12.504514 14683 raft_consensus.cc:2229] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:38:12.504866 14683 raft_consensus.cc:2258] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:12.504909 14682 tablet_replica.cc:330] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: stopping tablet replica
I20260926 11:38:12.505271 14682 raft_consensus.cc:2229] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:38:12.505594 14682 raft_consensus.cc:2258] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:12.507179 14682 ts_tablet_manager.cc:1905] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:12.507369 14683 ts_tablet_manager.cc:1905] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:12.510193 14682 ts_tablet_manager.cc:1918] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:12.510330 14682 log.cc:1196] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/da369d47d95a4bd4b77185b2fe7b766f
I20260926 11:38:12.510667 14682 ts_tablet_manager.cc:1939] T da369d47d95a4bd4b77185b2fe7b766f P 3b6557ae601448db8938555056bb09dd: Deleting consensus metadata
I20260926 11:38:12.510931 14683 ts_tablet_manager.cc:1918] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:12.511044 14683 log.cc:1196] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/da369d47d95a4bd4b77185b2fe7b766f
I20260926 11:38:12.511359 14683 ts_tablet_manager.cc:1939] T da369d47d95a4bd4b77185b2fe7b766f P 973fe1bffa2f428a936f2b9215650bb1: Deleting consensus metadata
W20260926 11:38:12.511838 14169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34453 (error 108)
I20260926 11:38:12.512655 14168 catalog_manager.cc:4841] TS 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639): tablet da369d47d95a4bd4b77185b2fe7b766f (table test-table [id=9185cffdece545f3a7ac6f8684fe520e]) successfully deleted
I20260926 11:38:12.512940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 14378
I20260926 11:38:12.512966 14169 catalog_manager.cc:4841] TS 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307): tablet da369d47d95a4bd4b77185b2fe7b766f (table test-table [id=9185cffdece545f3a7ac6f8684fe520e]) successfully deleted
I20260926 11:38:12.514118 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.513422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34453 (local address 127.0.0.1:47334)
0926 11:38:12.513974 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:12.514039 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448}
W20260926 11:38:12.514293 14169 catalog_manager.cc:4584] TS dd0f2c6e00c24e52b7b1078dc836a17a (127.0.98.1:34453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da369d47d95a4bd4b77185b2fe7b766f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34453: connect: Connection refused (error 111)
I20260926 11:38:12.519393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 14512
I20260926 11:38:12.526291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 14150
I20260926 11:38:12.534168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40241
--webserver_interface=127.0.98.62
--webserver_port=32769
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:12.658128 14685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:12.658465 14685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:12.658512 14685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:12.662822 14685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:12.662907 14685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:12.662930 14685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:12.662951 14685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:12.662969 14685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:12.668712 14685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:12.669883 14685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:12.670933 14685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:12.686499 14694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:12.686437 14691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:12.686450 14692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:12.687073 14685 server_base.cc:1034] running on GCE node
I20260926 11:38:12.687834 14685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:12.689298 14685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:12.690526 14685 hybrid_clock.cc:648] HybridClock initialized: now 1790422692690512 us; error 31 us; skew 500 ppm
Sep 26 11:38:12 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:12.694129 14685 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:38:12.695647 14685 webserver.cc:460] Webserver started at http://127.0.98.62:32769/ using document root <none> and password file <none>
I20260926 11:38:12.696352 14685 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:12.696446 14685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:12.701339 14685 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:38:12.704221 14701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:12.705956 14685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:38:12.706125 14685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "a73eedb1dbc52ba0cc5344736ab89f31"
server_key_iv: "5a589e33382780ac9ae64fc38288e1aa"
server_key_version: "encryptionkey@0"
I20260926 11:38:12.706790 14685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:12.719149 14685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:12.722909 14685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:12.723107 14685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:12.731464 14685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40241
I20260926 11:38:12.732019 14753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40241 every 8 connection(s)
I20260926 11:38:12.732877 14685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:38:12.738080 14754 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:38:12.740309 14754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Bootstrap starting.
I20260926 11:38:12.742162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 14685
I20260926 11:38:12.742744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34453
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40241
--webserver_port=37933
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:38:12.766099 14754 log.cc:824] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:12.778072 14754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:38:12.778611 14754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Bootstrap complete.
I20260926 11:38:12.784377 14754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:12.784937 14754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1888e607c9fb48bf98ac8f01bb8d7d0b, State: Initialized, Role: FOLLOWER
I20260926 11:38:12.785557 14754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:12.785718 14754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:38:12.785791 14754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:38:12.785892 14754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:38:12.788707 14754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:12.789119 14754 leader_election.cc:302] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1888e607c9fb48bf98ac8f01bb8d7d0b; no voters:
I20260926 11:38:12.789482 14754 leader_election.cc:288] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 11:38:12.789670 14758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 2 FOLLOWER]: Leader election won for term 2
I20260926 11:38:12.789958 14758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 2 LEADER]: Becoming Leader. State: Replica: 1888e607c9fb48bf98ac8f01bb8d7d0b, State: Running, Role: LEADER
I20260926 11:38:12.790300 14758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:12.790885 14754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:38:12.791977 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } } }
I20260926 11:38:12.792106 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:12.792034 14760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1888e607c9fb48bf98ac8f01bb8d7d0b. Latest consensus state: current_term: 2 leader_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } } }
I20260926 11:38:12.792428 14760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:12.792743 14762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:38:12.798127 14762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:38:12.799101 14762 catalog_manager.cc:1259] Loaded cluster ID: a2b017d2ba50468d90ab90e53a183d09
I20260926 11:38:12.799189 14762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:38:12.800395 14762 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 11:38:12.803182 14762 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 11:38:12.812825 14762 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:38:12.813695 14762 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Loaded TSK: 0
I20260926 11:38:12.814750 14762 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:38:12.889225 14756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:12.889611 14756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:12.889694 14756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:12.894520 14756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:12.894631 14756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:12.894730 14756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:38:12.900609 14756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37933
--tserver_master_addrs=127.0.98.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:12.902036 14756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:12.903118 14756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:12.913689 14781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:12.913866 14756 server_base.cc:1034] running on GCE node
W20260926 11:38:12.914043 14784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:12.913689 14782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:12.914781 14756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:12.915501 14756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:12.916731 14756 hybrid_clock.cc:648] HybridClock initialized: now 1790422692916698 us; error 50 us; skew 500 ppm
Sep 26 11:38:12 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:12.919651 14756 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:38:12.920717 14756 webserver.cc:460] Webserver started at http://127.0.98.1:37933/ using document root <none> and password file <none>
I20260926 11:38:12.921291 14756 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:12.921361 14756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:12.925596 14756 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:38:12.928083 14791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:12.929476 14756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:12.929615 14756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "dd0f2c6e00c24e52b7b1078dc836a17a"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "799c95ce2842d3bca736a28598a2b38b"
server_key_iv: "7e5d9f7264b518baaaab30bf6ed001e6"
server_key_version: "encryptionkey@0"
I20260926 11:38:12.930210 14756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:12.946026 14756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:12.949662 14756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:12.949893 14756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:12.950585 14756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:12.951810 14798 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:38:12.957595 14756 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:38:12.957715 14756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260926 11:38:12.957805 14756 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:38:12.959808 14756 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:38:12.959895 14756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260926 11:38:12.960230 14798 tablet_bootstrap.cc:492] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Bootstrap starting.
I20260926 11:38:12.970737 14756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34453
I20260926 11:38:12.970765 14905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34453 every 8 connection(s)
I20260926 11:38:12.971951 14756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:38:12.981642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 14756
I20260926 11:38:12.982105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45307
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40241
--webserver_port=38211
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:38:12.982882 14798 log.cc:824] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:38:12 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:12.992313 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.974118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36173 (local address 127.0.98.62:40241)
0926 11:38:12.974514 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:12.974534 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:12.975078 (+ 544us) server_negotiation.cc:407] Connection header received
0926 11:38:12.976754 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:12.976764 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:12.977083 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:12.977368 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:12.979203 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.980528 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.982322 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.983132 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.983332 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.986484 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:12.986493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:12.986512 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:12.986554 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:12.988642 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:12.989276 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:12.989282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:12.989286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:12.989425 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:12.989790 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:12.989794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:12.989796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:12.990169 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:12.990322 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:12.990872 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 11:38:12.991157 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
I20260926 11:38:12.992337 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:12.973992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:36173)
0926 11:38:12.974870 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:12.974933 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:12.976273 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:12.977537 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:12.977549 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:12.978091 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:12.978955 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.978983 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.980736 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.980741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.982139 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:12.982153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:12.983561 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:12.983567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:12.983832 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:12.984616 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:12.984639 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:12.986208 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:12.988811 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:12.988817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:12.988831 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:12.989132 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:12.989536 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:12.989540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:12.989541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:12.989684 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:12.990291 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:12.990297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:12.990569 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 11:38:12.990762 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 11:38:12.991140 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":152,"threads_started":1}
I20260926 11:38:12.994217 14798 tablet_bootstrap.cc:492] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:38:12.994762 14798 tablet_bootstrap.cc:492] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Bootstrap complete.
I20260926 11:38:12.995117 14906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40241
I20260926 11:38:12.995359 14906 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:12.995934 14906 heartbeater.cc:511] Master 127.0.98.62:40241 requested a full tablet report, sending...
I20260926 11:38:12.996315 14798 ts_tablet_manager.cc:1397] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.009s
I20260926 11:38:12.998508 14716 ts_manager.cc:194] Registered new tserver with Master: dd0f2c6e00c24e52b7b1078dc836a17a (127.0.98.1:34453)
I20260926 11:38:12.999415 14798 raft_consensus.cc:348] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:13.000013 14798 raft_consensus.cc:729] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0f2c6e00c24e52b7b1078dc836a17a, State: Initialized, Role: FOLLOWER
I20260926 11:38:13.000524 14798 consensus_queue.cc:260] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } } peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } }
I20260926 11:38:13.001585 14798 ts_tablet_manager.cc:1428] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:38:13.002751 14716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36173
I20260926 11:38:13.005101 14906 heartbeater.cc:503] Master 127.0.98.62:40241 was elected leader, sending a full tablet report...
Sep 26 11:38:13 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:38:13.015877 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.002020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.0.98.1:34453)
0926 11:38:13.002330 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.002342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.002705 (+ 363us) server_negotiation.cc:407] Connection header received
0926 11:38:13.002937 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.002943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.003149 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.003295 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:13.005599 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.006664 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.008128 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.008571 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.008680 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.011349 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:13.011359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:13.011370 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:13.011397 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:13.013344 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:13.014021 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:13.014035 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:13.014036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:13.014137 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:13.014531 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:13.014535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:13.014538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:13.014850 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:13.015016 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.015444 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.015681 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260926 11:38:13.015877 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.001720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34453 (local address 127.0.0.1:47338)
0926 11:38:13.002545 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:13.002589 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:13.002734 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:13.003319 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:13.003324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:13.005274 (+ 1950us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:13.005447 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.005456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.006925 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.006928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.007958 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.007971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.008695 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.008717 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.008835 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.009445 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:13.009460 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:13.011179 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:13.013527 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:13.013543 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:13.013550 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:13.013864 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:13.014263 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:13.014267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:13.014269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:13.014400 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:13.015000 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:13.015008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:13.015160 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 11:38:13.015332 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 11:38:13.015627 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":114,"threads_started":1}
I20260926 11:38:13.017136 14840 tablet_service.cc:1514] Processing DeleteTablet for tablet da369d47d95a4bd4b77185b2fe7b766f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47338
I20260926 11:38:13.018311 14915 tablet_replica.cc:330] stopping tablet replica
I20260926 11:38:13.018694 14915 raft_consensus.cc:2229] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:38:13.018980 14915 raft_consensus.cc:2258] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:13.020212 14915 ts_tablet_manager.cc:1905] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:13.023936 14915 ts_tablet_manager.cc:1918] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:13.024060 14915 log.cc:1196] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/da369d47d95a4bd4b77185b2fe7b766f
I20260926 11:38:13.024333 14915 ts_tablet_manager.cc:1939] T da369d47d95a4bd4b77185b2fe7b766f P dd0f2c6e00c24e52b7b1078dc836a17a: Deleting consensus metadata
I20260926 11:38:13.026182 14705 catalog_manager.cc:4841] TS dd0f2c6e00c24e52b7b1078dc836a17a (127.0.98.1:34453): tablet da369d47d95a4bd4b77185b2fe7b766f (table test-table [id=9185cffdece545f3a7ac6f8684fe520e]) successfully deleted
W20260926 11:38:13.115489 14910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:13.115823 14910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:13.115868 14910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:13.120615 14910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:13.120704 14910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:13.120838 14910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:38:13.127553 14910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38211
--tserver_master_addrs=127.0.98.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:13.128978 14910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:13.130168 14910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:13.141412 14923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:13.141407 14920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:13.141562 14910 server_base.cc:1034] running on GCE node
W20260926 11:38:13.141466 14921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:13.142220 14910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:13.143076 14910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:13.144882 14910 hybrid_clock.cc:648] HybridClock initialized: now 1790422693144859 us; error 108 us; skew 500 ppm
Sep 26 11:38:13 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:13.148404 14910 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:38:13.149359 14910 webserver.cc:460] Webserver started at http://127.0.98.2:38211/ using document root <none> and password file <none>
I20260926 11:38:13.149931 14910 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:13.150003 14910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:13.154104 14910 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:38:13.156502 14930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:13.158124 14910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:38:13.158334 14910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "973fe1bffa2f428a936f2b9215650bb1"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "2266b4ab3cd1d86d76de9f3749b7b390"
server_key_iv: "ef301027c613473ce3905b62118407a2"
server_key_version: "encryptionkey@0"
I20260926 11:38:13.158994 14910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:13.184226 14910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:13.188120 14910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:13.188349 14910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:13.189033 14910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:13.189961 14910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:13.190038 14910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:13.190116 14910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:13.190173 14910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:13.203157 14910 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45307
I20260926 11:38:13.203207 15043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45307 every 8 connection(s)
I20260926 11:38:13.204320 14910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:38:13.211024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 14910
I20260926 11:38:13.211788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41639
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40241
--webserver_port=42541
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:38:13 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:13.221066 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.206289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58663 (local address 127.0.98.62:40241)
0926 11:38:13.206445 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.206449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.207345 (+ 896us) server_negotiation.cc:407] Connection header received
0926 11:38:13.208884 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.208888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.208955 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.209060 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:13.210440 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.211326 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.213042 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.213310 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.213446 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.216577 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:13.216581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:13.216583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:13.216613 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:13.218362 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:13.219030 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:13.219035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:13.219036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:13.219114 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:13.219504 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:13.219507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:13.219509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:13.219856 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:13.219959 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.220782 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.220933 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 11:38:13.222431 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.206720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:58663)
0926 11:38:13.207178 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:13.207221 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:13.208615 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:13.209241 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:13.209248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:13.209706 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:13.210247 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.210262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.211729 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.211732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.212883 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.212897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.213501 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.213506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.213705 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.214387 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:13.214405 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:13.216346 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:13.218532 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:13.218539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:13.218558 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:13.218898 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:13.219220 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:13.219223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:13.219225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:13.219393 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:13.219993 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:13.220002 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:13.220438 (+ 436us) client_negotiation.cc:769] Sending connection context
0926 11:38:13.220705 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 11:38:13.221156 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260926 11:38:13.224428 15044 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40241
I20260926 11:38:13.224821 15044 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:13.225419 15044 heartbeater.cc:511] Master 127.0.98.62:40241 requested a full tablet report, sending...
I20260926 11:38:13.226859 14716 ts_manager.cc:194] Registered new tserver with Master: 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307)
I20260926 11:38:13.227872 14716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58663
W20260926 11:38:13.338318 15047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:13.338601 15047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:13.338652 15047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:13.343361 15047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:13.343464 15047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:13.343536 15047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:38:13.349380 15047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42541
--tserver_master_addrs=127.0.98.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:13.350495 15047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:13.351491 15047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:13.362226 15055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:13.362226 15052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:13.362385 15047 server_base.cc:1034] running on GCE node
W20260926 11:38:13.362591 15053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:13.363089 15047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:13.364077 15047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:13.365283 15047 hybrid_clock.cc:648] HybridClock initialized: now 1790422693365236 us; error 61 us; skew 500 ppm
Sep 26 11:38:13 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:13.368772 15047 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:38:13.369757 15047 webserver.cc:460] Webserver started at http://127.0.98.3:42541/ using document root <none> and password file <none>
I20260926 11:38:13.370350 15047 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:13.370432 15047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:13.374753 15047 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:38:13.377221 15062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:13.378759 15047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:38:13.378886 15047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "3b6557ae601448db8938555056bb09dd"
format_stamp: "Formatted at 2026-09-26 11:38:12 on dist-test-slave-4pw5"
server_key: "ea6ab89e8b72633a78861989fae8d8b9"
server_key_iv: "aa8601a81413f9b00c6dcb2732e2e558"
server_key_version: "encryptionkey@0"
I20260926 11:38:13.379549 15047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:13.394429 15047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:13.397956 15047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:13.398134 15047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:13.398718 15047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:13.399560 15047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:13.399685 15047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:13.399730 15047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:13.399751 15047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:13.409430 15047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41639
I20260926 11:38:13.409454 15175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41639 every 8 connection(s)
I20260926 11:38:13.410629 15047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:38:13.419663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 15047
Sep 26 11:38:13 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:13.427026 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.412758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55549 (local address 127.0.98.62:40241)
0926 11:38:13.412950 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.412954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.413894 (+ 940us) server_negotiation.cc:407] Connection header received
0926 11:38:13.415206 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.415209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.415284 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.415380 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:13.416802 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.417331 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.418504 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.418723 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.418817 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.422519 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:13.422523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:13.422525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:13.422552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:13.424417 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:13.425143 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:13.425148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:13.425151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:13.425249 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:13.425625 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:13.425631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:13.425635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:13.425882 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:13.426001 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.426728 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.426866 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 11:38:13.428287 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.413213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:55549)
0926 11:38:13.413722 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:13.413767 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:13.414980 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:13.415684 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:13.415692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:13.416120 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:13.416622 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.416636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.417463 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.417466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.418358 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.418368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.418910 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.418914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.419084 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.420046 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:13.420068 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:13.422303 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:13.424553 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:13.424564 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:13.424588 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:13.425000 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:13.425356 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:13.425359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:13.425360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:13.425516 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:13.425997 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:13.426002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:13.426405 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 11:38:13.426646 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 11:38:13.427066 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":156,"threads_started":1}
I20260926 11:38:13.429669 15176 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40241
I20260926 11:38:13.429965 15176 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:13.430311 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.420467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.0.98.62:40241)
0926 11:38:13.420847 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.420856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.420898 (+ 42us) server_negotiation.cc:407] Connection header received
0926 11:38:13.420954 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.420958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.421020 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.421108 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:13.422152 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.422868 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.424233 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.424497 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.424607 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.426144 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:13.426147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:13.426150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:13.426174 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:13.428108 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:13.428811 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:13.428816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:13.428818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:13.428925 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:13.429291 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:13.429295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:13.429296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:13.429560 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:13.429791 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.429966 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.430119 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
I20260926 11:38:13.430714 15176 heartbeater.cc:511] Master 127.0.98.62:40241 requested a full tablet report, sending...
I20260926 11:38:13.431860 14717 ts_manager.cc:194] Registered new tserver with Master: 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639)
I20260926 11:38:13.432543 14717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55549
I20260926 11:38:13.445051 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:38:13.450424 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.445809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.0.98.62:40241)
0926 11:38:13.446010 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.446016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.446304 (+ 288us) server_negotiation.cc:407] Connection header received
0926 11:38:13.446416 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.446418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.446532 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.446631 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:38:13.446903 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.447655 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.448775 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.449025 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.449120 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.449482 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:38:13.449942 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:38:13.450044 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.450139 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.450209 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 11:38:13.451282 14717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:38:13.454811 14717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:38:13.464327 14840 tablet_service.cc:1467] Processing CreateTablet for tablet 24d008b2da2d4ae888e8ff660e65f733 (DEFAULT_TABLE table=test-table [id=152ab193b2bb448097dc4d45af679025]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:13.465169 14840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d008b2da2d4ae888e8ff660e65f733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:13.468745 14798 tablet_bootstrap.cc:492] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Bootstrap starting.
I20260926 11:38:13.469540 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.463197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45307 (local address 127.0.0.1:38720)
0926 11:38:13.463343 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:13.463360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:13.463608 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:13.464646 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:13.464650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:13.464670 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:13.464805 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.464812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.466491 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.466495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.467963 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.467976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.469090 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.469094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.469201 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.469246 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 11:38:13.469305 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 11:38:13.469398 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":28}
I20260926 11:38:13.469882 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.463829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41639 (local address 127.0.0.1:58812)
0926 11:38:13.464106 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:13.464123 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:13.464314 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:13.465141 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:13.465145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:13.465167 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:13.465422 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.465431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.466969 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.466976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.467926 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.467943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.469355 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.469358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.469445 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.469606 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 11:38:13.469672 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 11:38:13.469762 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":22,"thread_start_us":106,"threads_started":1}
I20260926 11:38:13.469951 14798 tablet_bootstrap.cc:654] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:13.470281 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.463398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.0.98.2:45307)
0926 11:38:13.464080 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.464096 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.464123 (+ 27us) server_negotiation.cc:407] Connection header received
0926 11:38:13.464210 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.464216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.464463 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.464973 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:13.465167 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.466320 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.468119 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.468930 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.469792 (+ 862us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.469963 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.470025 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.470140 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":198,"threads_started":1}
I20260926 11:38:13.470410 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.464248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.0.98.3:41639)
0926 11:38:13.464668 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.464679 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.464700 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:38:13.464771 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.464777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.464965 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.465127 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:13.465581 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.466825 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.468062 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.468930 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.469061 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.470070 (+ 1009us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.470123 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.470268 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":117,"threads_started":1}
I20260926 11:38:13.472136 14798 tablet_bootstrap.cc:492] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: No bootstrap required, opened a new log
I20260926 11:38:13.472242 14798 ts_tablet_manager.cc:1397] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 11:38:13.472723 14798 raft_consensus.cc:348] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:13.472887 14798 raft_consensus.cc:374] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:13.472926 14798 raft_consensus.cc:729] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0f2c6e00c24e52b7b1078dc836a17a, State: Initialized, Role: FOLLOWER
I20260926 11:38:13.473089 14798 consensus_queue.cc:260] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:13.473347 14798 ts_tablet_manager.cc:1428] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:38:13.473572 15110 tablet_service.cc:1467] Processing CreateTablet for tablet 24d008b2da2d4ae888e8ff660e65f733 (DEFAULT_TABLE table=test-table [id=152ab193b2bb448097dc4d45af679025]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:13.473569 14969 tablet_service.cc:1467] Processing CreateTablet for tablet 24d008b2da2d4ae888e8ff660e65f733 (DEFAULT_TABLE table=test-table [id=152ab193b2bb448097dc4d45af679025]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:13.474782 14969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d008b2da2d4ae888e8ff660e65f733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:13.474860 15110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d008b2da2d4ae888e8ff660e65f733. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 11:38:13.476267 14907 tablet.cc:2369] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:38:13.482591 15190 tablet_bootstrap.cc:492] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: Bootstrap starting.
I20260926 11:38:13.483183 15189 tablet_bootstrap.cc:492] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Bootstrap starting.
I20260926 11:38:13.485307 15190 tablet_bootstrap.cc:654] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:13.485687 15189 tablet_bootstrap.cc:654] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:13.486426 15190 log.cc:824] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:13.486426 15189 log.cc:824] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:13.488437 15190 tablet_bootstrap.cc:492] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: No bootstrap required, opened a new log
I20260926 11:38:13.488471 15189 tablet_bootstrap.cc:492] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: No bootstrap required, opened a new log
I20260926 11:38:13.488651 15190 ts_tablet_manager.cc:1397] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:38:13.488698 15189 ts_tablet_manager.cc:1397] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:38:13.491956 15190 raft_consensus.cc:348] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:13.491957 15189 raft_consensus.cc:348] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:13.492290 15189 raft_consensus.cc:374] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:13.492282 15190 raft_consensus.cc:374] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:13.492353 15190 raft_consensus.cc:729] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6557ae601448db8938555056bb09dd, State: Initialized, Role: FOLLOWER
I20260926 11:38:13.492354 15189 raft_consensus.cc:729] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973fe1bffa2f428a936f2b9215650bb1, State: Initialized, Role: FOLLOWER
I20260926 11:38:13.492849 15190 consensus_queue.cc:260] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:13.492851 15189 consensus_queue.cc:260] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:13.493592 15176 heartbeater.cc:503] Master 127.0.98.62:40241 was elected leader, sending a full tablet report...
I20260926 11:38:13.493592 15044 heartbeater.cc:503] Master 127.0.98.62:40241 was elected leader, sending a full tablet report...
I20260926 11:38:13.493911 15189 ts_tablet_manager.cc:1428] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:38:13.493912 15190 ts_tablet_manager.cc:1428] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:38:13.502082 15193 raft_consensus.cc:482] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:13.502295 15193 raft_consensus.cc:504] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:13.503492 15193 leader_election.cc:288] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307), dd0f2c6e00c24e52b7b1078dc836a17a (127.0.98.1:34453)
I20260926 11:38:13.507365 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.503723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45307 (local address 127.0.98.3:53847)
0926 11:38:13.503880 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:13.503897 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:13.504003 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:13.504398 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:13.504403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:13.504424 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:13.504591 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.504597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.505442 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.505448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.506255 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.506263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.506934 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.506937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.507052 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.507070 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 11:38:13.507158 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 11:38:13.507235 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"spinlock_wait_cycles":75008}
I20260926 11:38:13.507365 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.503813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53847 (local address 127.0.98.2:45307)
0926 11:38:13.503958 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.503963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.503980 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:38:13.504146 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.504150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.504206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.504276 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:13.504715 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.505325 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.506383 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.506801 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.506925 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.506989 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.507173 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.507241 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:38:13.507831 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.503985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34453 (local address 127.0.98.3:45869)
0926 11:38:13.504337 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:13.504350 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:13.504480 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:13.504799 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:13.504802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:13.504816 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:13.504939 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.504945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.505759 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.505765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.506547 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:13.506555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.507263 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.507267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:13.507357 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.507377 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:38:13.507621 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 11:38:13.507694 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
I20260926 11:38:13.507861 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.504175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45869 (local address 127.0.98.1:34453)
0926 11:38:13.504297 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.504301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.504432 (+ 131us) server_negotiation.cc:407] Connection header received
0926 11:38:13.504595 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.504598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.504674 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.504791 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:13.505117 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.505643 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.506677 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.507144 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.507267 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.507450 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.507652 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.507749 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:38:13.508211 14993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d008b2da2d4ae888e8ff660e65f733" candidate_uuid: "3b6557ae601448db8938555056bb09dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973fe1bffa2f428a936f2b9215650bb1" is_pre_election: true
I20260926 11:38:13.508481 14860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d008b2da2d4ae888e8ff660e65f733" candidate_uuid: "3b6557ae601448db8938555056bb09dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" is_pre_election: true
I20260926 11:38:13.508586 14993 raft_consensus.cc:2454] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6557ae601448db8938555056bb09dd in term 0.
I20260926 11:38:13.508738 14860 raft_consensus.cc:2454] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6557ae601448db8938555056bb09dd in term 0.
I20260926 11:38:13.509084 15066 leader_election.cc:302] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6557ae601448db8938555056bb09dd, 973fe1bffa2f428a936f2b9215650bb1; no voters:
I20260926 11:38:13.509337 15193 raft_consensus.cc:2781] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:38:13.509415 15193 raft_consensus.cc:482] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:38:13.509445 15193 raft_consensus.cc:3037] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:13.510612 15193 raft_consensus.cc:504] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:13.510977 15193 leader_election.cc:288] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [CANDIDATE]: Term 1 election: Requested vote from peers 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307), dd0f2c6e00c24e52b7b1078dc836a17a (127.0.98.1:34453)
I20260926 11:38:13.511452 14860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d008b2da2d4ae888e8ff660e65f733" candidate_uuid: "3b6557ae601448db8938555056bb09dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a"
I20260926 11:38:13.511475 14993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d008b2da2d4ae888e8ff660e65f733" candidate_uuid: "3b6557ae601448db8938555056bb09dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "973fe1bffa2f428a936f2b9215650bb1"
I20260926 11:38:13.511655 14860 raft_consensus.cc:3037] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:13.511691 14993 raft_consensus.cc:3037] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:13.512753 14860 raft_consensus.cc:2454] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6557ae601448db8938555056bb09dd in term 1.
I20260926 11:38:13.512773 14993 raft_consensus.cc:2454] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6557ae601448db8938555056bb09dd in term 1.
I20260926 11:38:13.513094 15066 leader_election.cc:302] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6557ae601448db8938555056bb09dd, 973fe1bffa2f428a936f2b9215650bb1; no voters:
I20260926 11:38:13.513309 15193 raft_consensus.cc:2781] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:38:13.513553 15193 raft_consensus.cc:686] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 1 LEADER]: Becoming Leader. State: Replica: 3b6557ae601448db8938555056bb09dd, State: Running, Role: LEADER
I20260926 11:38:13.513928 15193 consensus_queue.cc:237] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:13.517184 14717 catalog_manager.cc:5495] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6557ae601448db8938555056bb09dd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3b6557ae601448db8938555056bb09dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } health_report { overall_health: UNKNOWN } } }
I20260926 11:38:13.537829 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.534071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.0.98.3:41639)
0926 11:38:13.534223 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:13.534226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:13.534241 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:38:13.534349 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:13.534351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:13.534411 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:13.534510 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:38:13.534818 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.535347 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.536279 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:13.536494 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:13.536574 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:13.536838 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:38:13.537336 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:38:13.537441 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:13.537586 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 11:38:13.537641 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:38:13.544425 14993 raft_consensus.cc:1261] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [term 1 FOLLOWER]: Refusing update from remote peer 3b6557ae601448db8938555056bb09dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:13.544422 14860 raft_consensus.cc:1261] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Refusing update from remote peer 3b6557ae601448db8938555056bb09dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:13.545255 15193 consensus_queue.cc:1035] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:13.545522 15196 consensus_queue.cc:1035] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:13.555759 15201 mvcc.cc:205] Tried to move back new op lower bound from 7333571352748875776 to 7333571352636059648. Current Snapshot: MvccSnapshot[applied={T|T < 7333571352748875776 or (T in {7333571352748875776})}]
I20260926 11:38:13.563041 14716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:13.563241 14716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:13.566250 14716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Sending DeleteTablet for 3 replicas of tablet 24d008b2da2d4ae888e8ff660e65f733
I20260926 11:38:13.567065 14840 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d008b2da2d4ae888e8ff660e65f733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47338
I20260926 11:38:13.567173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 14756
I20260926 11:38:13.567173 15110 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d008b2da2d4ae888e8ff660e65f733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58812
I20260926 11:38:13.567986 15209 tablet_replica.cc:330] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: stopping tablet replica
I20260926 11:38:13.568367 15209 raft_consensus.cc:2229] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:38:13.568634 14969 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d008b2da2d4ae888e8ff660e65f733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38720
I20260926 11:38:13.568850 15209 raft_consensus.cc:2258] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:13.569150 15210 tablet_replica.cc:330] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: stopping tablet replica
I20260926 11:38:13.569535 15210 raft_consensus.cc:2229] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:38:13.569911 15210 raft_consensus.cc:2258] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:13.570778 15209 ts_tablet_manager.cc:1905] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:13.571499 15210 ts_tablet_manager.cc:1905] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:13.574488 15210 ts_tablet_manager.cc:1918] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:13.574489 15209 ts_tablet_manager.cc:1918] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:13.574632 15209 log.cc:1196] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/24d008b2da2d4ae888e8ff660e65f733
I20260926 11:38:13.574787 15210 log.cc:1196] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/24d008b2da2d4ae888e8ff660e65f733
I20260926 11:38:13.575037 15209 ts_tablet_manager.cc:1939] T 24d008b2da2d4ae888e8ff660e65f733 P 3b6557ae601448db8938555056bb09dd: Deleting consensus metadata
W20260926 11:38:13.575210 14705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34453 (error 108)
I20260926 11:38:13.575515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 14910
I20260926 11:38:13.577281 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.576890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34453 (local address 127.0.0.1:47350)
0926 11:38:13.577113 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:13.577184 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 11:38:13.577744 14704 catalog_manager.cc:4841] TS 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639): tablet 24d008b2da2d4ae888e8ff660e65f733 (table test-table [id=152ab193b2bb448097dc4d45af679025]) successfully deleted
W20260926 11:38:13.577589 14705 catalog_manager.cc:4584] TS dd0f2c6e00c24e52b7b1078dc836a17a (127.0.98.1:34453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24d008b2da2d4ae888e8ff660e65f733: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34453: connect: Connection refused (error 111)
I20260926 11:38:13.582172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 15047
I20260926 11:38:13.582384 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:13.582079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45307 (local address 127.0.0.1:38728)
0926 11:38:13.582204 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:13.582276 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 11:38:13.589646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 14685
I20260926 11:38:13.596240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40241
--webserver_interface=127.0.98.62
--webserver_port=32769
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:13.724882 15212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:13.725404 15212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:13.725477 15212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:13.729969 15212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:13.730043 15212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:13.730063 15212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:13.730082 15212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:13.730096 15212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:13.735931 15212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:13.737139 15212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:13.738224 15212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:13.747670 15220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:13.747668 15217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:13.748136 15212 server_base.cc:1034] running on GCE node
W20260926 11:38:13.747689 15218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:13.748948 15212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:13.750202 15212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:13.751430 15212 hybrid_clock.cc:648] HybridClock initialized: now 1790422693751397 us; error 48 us; skew 500 ppm
Sep 26 11:38:13 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:13.754565 15212 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:38:13.755674 15212 webserver.cc:460] Webserver started at http://127.0.98.62:32769/ using document root <none> and password file <none>
I20260926 11:38:13.756285 15212 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:13.756387 15212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:13.760603 15212 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 11:38:13.762930 15227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:13.764402 15212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:38:13.764565 15212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "a73eedb1dbc52ba0cc5344736ab89f31"
server_key_iv: "5a589e33382780ac9ae64fc38288e1aa"
server_key_version: "encryptionkey@0"
I20260926 11:38:13.765184 15212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:13.796379 15212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:13.800060 15212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:13.800285 15212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:13.807446 15212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40241
I20260926 11:38:13.807783 15279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40241 every 8 connection(s)
I20260926 11:38:13.808621 15212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:38:13.813192 15280 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:38:13.814473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 15212
I20260926 11:38:13.814985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34453
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40241
--webserver_port=37933
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:38:13.815224 15280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Bootstrap starting.
I20260926 11:38:13.835014 15280 log.cc:824] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:13.846834 15280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:38:13.847302 15280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b: Bootstrap complete.
I20260926 11:38:13.852015 15280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:13.852546 15280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1888e607c9fb48bf98ac8f01bb8d7d0b, State: Initialized, Role: FOLLOWER
I20260926 11:38:13.853135 15280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:13.853299 15280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:38:13.853389 15280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:38:13.853519 15280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 2 FOLLOWER]: Advancing to term 3
I20260926 11:38:13.855692 15280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:13.856073 15280 leader_election.cc:302] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1888e607c9fb48bf98ac8f01bb8d7d0b; no voters:
I20260926 11:38:13.856459 15280 leader_election.cc:288] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 11:38:13.856552 15284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 3 FOLLOWER]: Leader election won for term 3
I20260926 11:38:13.856760 15284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [term 3 LEADER]: Becoming Leader. State: Replica: 1888e607c9fb48bf98ac8f01bb8d7d0b, State: Running, Role: LEADER
I20260926 11:38:13.857075 15284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } }
I20260926 11:38:13.857780 15280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:38:13.858593 15285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } } }
I20260926 11:38:13.858636 15286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1888e607c9fb48bf98ac8f01bb8d7d0b. Latest consensus state: current_term: 3 leader_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888e607c9fb48bf98ac8f01bb8d7d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40241 } } }
I20260926 11:38:13.858721 15285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:13.858753 15286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888e607c9fb48bf98ac8f01bb8d7d0b [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:13.859310 15291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:38:13.865314 15291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:38:13.866309 15291 catalog_manager.cc:1259] Loaded cluster ID: a2b017d2ba50468d90ab90e53a183d09
I20260926 11:38:13.866384 15291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 11:38:13.867607 15291 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f63263299ad google::LogMessage::Fail() at ??:0
@ 0x7f632632c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f63263294de google::LogMessage::Flush() at ??:0
@ 0x7f632632aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6330c68fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6330c69d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6330c66579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6330caa640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f632857b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f632857bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f632857d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f632856c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6329bb06db start_thread at ??:0
@ 0x7f6324e2761f clone at ??:0
W20260926 11:38:13.963874 15282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:13.964329 15282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:13.964417 15282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:13.971168 15282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:13.971293 15282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:13.971546 15282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:38:13.980652 15282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37933
--tserver_master_addrs=127.0.98.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:13.982138 15282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:13.983537 15282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:13.997256 15322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:13.997494 15282 server_base.cc:1034] running on GCE node
W20260926 11:38:13.997256 15321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:13.997638 15324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:13.998117 15282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:13.998994 15282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:14.000192 15282 hybrid_clock.cc:648] HybridClock initialized: now 1790422694000153 us; error 52 us; skew 500 ppm
Sep 26 11:38:13 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:14.003723 15282 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:38:14.004745 15282 webserver.cc:460] Webserver started at http://127.0.98.1:37933/ using document root <none> and password file <none>
I20260926 11:38:14.005376 15282 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:14.005481 15282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:14.010571 15282 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:38:14.013595 15331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:14.014956 15282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:38:14.015123 15282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "dd0f2c6e00c24e52b7b1078dc836a17a"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "799c95ce2842d3bca736a28598a2b38b"
server_key_iv: "7e5d9f7264b518baaaab30bf6ed001e6"
server_key_version: "encryptionkey@0"
I20260926 11:38:14.015848 15282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:14.035162 15282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:14.039105 15282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:14.039346 15282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:14.040084 15282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:14.041357 15338 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:38:14.047165 15282 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:38:14.047299 15282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 11:38:14.047529 15282 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:38:14.049578 15282 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:38:14.049674 15282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260926 11:38:14.050009 15338 tablet_bootstrap.cc:492] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Bootstrap starting.
I20260926 11:38:14.061093 15282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34453
I20260926 11:38:14.061372 15445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34453 every 8 connection(s)
I20260926 11:38:14.062225 15282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:38:14.063202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 15282
I20260926 11:38:14.063920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45307
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40241
--webserver_port=38211
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:38:14.073890 15338 log.cc:824] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:14.082266 15338 tablet_bootstrap.cc:492] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:38:14.082742 15338 tablet_bootstrap.cc:492] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Bootstrap complete.
I20260926 11:38:14.084257 15338 ts_tablet_manager.cc:1397] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20260926 11:38:14.087170 15338 raft_consensus.cc:348] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:14.087769 15338 raft_consensus.cc:729] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0f2c6e00c24e52b7b1078dc836a17a, State: Initialized, Role: FOLLOWER
I20260926 11:38:14.088649 15338 consensus_queue.cc:260] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:14.089951 15338 ts_tablet_manager.cc:1428] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260926 11:38:14.157763 15449 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:38:14.064852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:58263)
0926 11:38:14.065475 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:14.065534 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:14.066628 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:14.156941 (+ 90313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":204,"threads_started":1}
W20260926 11:38:14.158365 15446 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40241 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40241: Client connection negotiation failed: client connection to 127.0.98.62:40241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 11:38:14.159219 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:14.158940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:44981)
0926 11:38:14.159080 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:14.159149 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 11:38:14.159963 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:14.159743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:39465)
0926 11:38:14.159853 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:14.159910 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 11:38:14.160161 15446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:38:14.208400 15448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:14.208765 15448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:14.208868 15448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:14.213330 15448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:14.213438 15448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:14.213539 15448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:38:14.220113 15448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38211
--tserver_master_addrs=127.0.98.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:14.221514 15448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:14.222674 15448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:14.234290 15459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:14.234369 15456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:14.234290 15457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:14.234524 15448 server_base.cc:1034] running on GCE node
I20260926 11:38:14.235085 15448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:14.236152 15448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:14.237349 15448 hybrid_clock.cc:648] HybridClock initialized: now 1790422694237308 us; error 55 us; skew 500 ppm
Sep 26 11:38:14 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:14.240576 15448 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:38:14.241528 15448 webserver.cc:460] Webserver started at http://127.0.98.2:38211/ using document root <none> and password file <none>
I20260926 11:38:14.242153 15448 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:14.242251 15448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:14.246397 15448 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:38:14.248821 15466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:14.250267 15448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:38:14.250420 15448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "973fe1bffa2f428a936f2b9215650bb1"
format_stamp: "Formatted at 2026-09-26 11:38:11 on dist-test-slave-4pw5"
server_key: "2266b4ab3cd1d86d76de9f3749b7b390"
server_key_iv: "ef301027c613473ce3905b62118407a2"
server_key_version: "encryptionkey@0"
I20260926 11:38:14.251026 15448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:14.266258 15448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:14.269860 15448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:14.270095 15448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:14.270732 15448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:14.271886 15473 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 11:38:14.276235 15473 ts_tablet_manager.cc:1852] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 11:38:14.276708 15473 ts_tablet_manager.cc:1905] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:14.282229 15473 ts_tablet_manager.cc:1918] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:14.282375 15473 log.cc:1196] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/24d008b2da2d4ae888e8ff660e65f733
I20260926 11:38:14.282819 15473 ts_tablet_manager.cc:1939] T 24d008b2da2d4ae888e8ff660e65f733 P 973fe1bffa2f428a936f2b9215650bb1: Deleting consensus metadata
I20260926 11:38:14.283664 15448 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 11:38:14.283771 15448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.000s
I20260926 11:38:14.283878 15448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:14.283926 15448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:14.294770 15448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45307
I20260926 11:38:14.294823 15580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45307 every 8 connection(s)
I20260926 11:38:14.295907 15448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:38:14.300334 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:14.298747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:57769)
0926 11:38:14.299361 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:14.299503 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":151,"threads_started":1}
W20260926 11:38:14.300833 15581 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40241 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40241: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
I20260926 11:38:14.301512 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:14.301278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:55045)
0926 11:38:14.301398 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:14.301455 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 11:38:14.302127 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:14.301914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:57123)
0926 11:38:14.302028 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:14.302073 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 11:38:14.302309 15581 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 11:38:14.303166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 15448
I20260926 11:38:14.303656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41639
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40241
--webserver_port=42541
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:14.425561 15584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:14.425866 15584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:14.425912 15584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:14.430390 15584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:14.430462 15584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:14.430527 15584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:38:14.436695 15584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42541
--tserver_master_addrs=127.0.98.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:14.437891 15584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:14.438905 15584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:14.449219 15592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:14.449219 15589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:14.449222 15590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:14.449358 15584 server_base.cc:1034] running on GCE node
I20260926 11:38:14.450083 15584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:14.450978 15584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:14.452217 15584 hybrid_clock.cc:648] HybridClock initialized: now 1790422694452181 us; error 79 us; skew 500 ppm
Sep 26 11:38:14 dist-test-slave-4pw5 krb5kdc[14137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:14.455860 15584 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:38:14.456900 15584 webserver.cc:460] Webserver started at http://127.0.98.3:42541/ using document root <none> and password file <none>
I20260926 11:38:14.457522 15584 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:14.457590 15584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:14.462044 15584 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:38:14.464447 15599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:14.465842 15584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:14.465988 15584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "3b6557ae601448db8938555056bb09dd"
format_stamp: "Formatted at 2026-09-26 11:38:12 on dist-test-slave-4pw5"
server_key: "ea6ab89e8b72633a78861989fae8d8b9"
server_key_iv: "aa8601a81413f9b00c6dcb2732e2e558"
server_key_version: "encryptionkey@0"
I20260926 11:38:14.466550 15584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:14.484344 15584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:14.487538 15584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:14.487802 15584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:14.488396 15584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:14.489298 15584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:14.489346 15584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:14.489387 15584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:14.489490 15584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:14.499120 15584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41639
I20260926 11:38:14.499141 15712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41639 every 8 connection(s)
I20260926 11:38:14.500290 15584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:38:14.501317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 15584
I20260926 11:38:14.504779 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:14.502894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:57207)
0926 11:38:14.503366 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:14.503478 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":129,"threads_started":1}
W20260926 11:38:14.505303 15713 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40241 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40241: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
I20260926 11:38:14.505898 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:14.505700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:32791)
0926 11:38:14.505804 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:14.505846 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 11:38:14.506443 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:14.506280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:55829)
0926 11:38:14.506364 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:14.506401 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 11:38:14.506598 15713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 11:38:15.161602 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:15.160966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:48409)
0926 11:38:15.161347 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:15.161436 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":83,"threads_started":1}
I20260926 11:38:15.303985 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:15.303159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:33785)
0926 11:38:15.303708 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:15.303829 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":109,"threads_started":1}
I20260926 11:38:15.508206 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:15.507374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:59391)
0926 11:38:15.507942 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:15.508046 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":113,"threads_started":1}
I20260926 11:38:15.883426 15720 raft_consensus.cc:482] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:15.883737 15720 raft_consensus.cc:504] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:15.885001 15720 leader_election.cc:288] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307), 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639)
Sep 26 11:38:15 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 11:38:15 dist-test-slave-4pw5 krb5kdc[14137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 11:38:15.913118 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:15.885502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41639 (local address 127.0.98.1:45155)
0926 11:38:15.886104 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:15.886119 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:15.886278 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:15.888587 (+ 2309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:15.888590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:15.889465 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:15.890603 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:15.890633 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:15.893490 (+ 2857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:15.893495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:15.895559 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:15.895604 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:15.897397 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:15.897403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:15.897862 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:15.903259 (+ 5397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:15.903300 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:15.906222 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:15.909510 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:15.909522 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:15.909546 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:15.909952 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:15.910449 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:15.910453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:15.910455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:15.910659 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:15.911686 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:15.911696 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:15.912195 (+ 499us) client_negotiation.cc:769] Sending connection context
0926 11:38:15.912391 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 11:38:15.912877 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":730,"spinlock_wait_cycles":27136,"thread_start_us":88,"threads_started":1}
I20260926 11:38:15.913281 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:15.885888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38783 (local address 127.0.98.2:45307)
0926 11:38:15.886420 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:15.886431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:15.886469 (+ 38us) server_negotiation.cc:407] Connection header received
0926 11:38:15.887331 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:15.887338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:15.887853 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:15.888256 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:15.890967 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:15.893197 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:15.896174 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:15.897197 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:15.897391 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:15.906894 (+ 9503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:15.906901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:15.906918 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:15.906966 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:15.909443 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:15.910230 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:15.910238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:15.910242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:15.910443 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:15.910823 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:15.910825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:15.910827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:15.911551 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:15.911740 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:15.912585 (+ 845us) server_negotiation.cc:299] Negotiation successful
0926 11:38:15.913046 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":156,"threads_started":1}
I20260926 11:38:15.913318 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:15.885698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45155 (local address 127.0.98.3:41639)
0926 11:38:15.886218 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:15.886251 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:15.886313 (+ 62us) server_negotiation.cc:407] Connection header received
0926 11:38:15.887313 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:15.887324 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:15.887853 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:15.888256 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:15.890967 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:15.893197 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:15.895848 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:15.897134 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:15.897391 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:15.906674 (+ 9283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:15.906687 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:15.906720 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:15.906776 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:15.909304 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:15.910110 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:15.910121 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:15.910132 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:15.910309 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:15.910784 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:15.910789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:15.910791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:15.911497 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:15.911690 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:15.912736 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0926 11:38:15.913096 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":108,"threads_started":1}
I20260926 11:38:15.913936 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:15.885502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45307 (local address 127.0.98.1:38783)
0926 11:38:15.886035 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:15.886055 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:15.886214 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:15.888560 (+ 2346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:15.888571 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:15.889465 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:15.890603 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:15.890633 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:15.893489 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:15.893495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:15.895558 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:15.895613 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:15.897397 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:15.897403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:15.897862 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:15.903259 (+ 5397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:15.903305 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:15.906559 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:15.909639 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:15.909643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:15.909646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:15.910057 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:15.910577 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:15.910580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:15.910582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:15.910682 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:15.911779 (+ 1097us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:15.911782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:15.912138 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 11:38:15.912391 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 11:38:15.913024 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":655,"spinlock_wait_cycles":57600,"thread_start_us":118,"threads_started":1}
W20260926 11:38:15.916855 15335 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
W20260926 11:38:15.917085 15334 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
I20260926 11:38:15.917263 15334 leader_election.cc:302] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd0f2c6e00c24e52b7b1078dc836a17a; no voters: 3b6557ae601448db8938555056bb09dd, 973fe1bffa2f428a936f2b9215650bb1
I20260926 11:38:15.917652 15720 raft_consensus.cc:2726] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:38:16.163017 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:16.162689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:43905)
0926 11:38:16.162844 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:16.162904 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 11:38:16.305822 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:16.305235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:42191)
0926 11:38:16.305595 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:16.305670 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":124,"threads_started":1}
I20260926 11:38:16.509858 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:16.509335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:36639)
0926 11:38:16.509664 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:16.509738 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
I20260926 11:38:17.164717 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:17.164117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:42837)
0926 11:38:17.164489 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:17.164574 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":127,"threads_started":1}
I20260926 11:38:17.307708 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:17.306977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:56319)
0926 11:38:17.307362 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:17.307458 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":95,"threads_started":1}
I20260926 11:38:17.511845 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:17.511053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:39405)
0926 11:38:17.511529 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:17.511707 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":113,"threads_started":1}
I20260926 11:38:17.932395 15730 raft_consensus.cc:482] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:17.932548 15730 raft_consensus.cc:504] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:17.933015 15730 leader_election.cc:288] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307), 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639)
W20260926 11:38:17.933939 15334 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
W20260926 11:38:17.934067 15335 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
I20260926 11:38:17.934224 15335 leader_election.cc:302] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd0f2c6e00c24e52b7b1078dc836a17a; no voters: 3b6557ae601448db8938555056bb09dd, 973fe1bffa2f428a936f2b9215650bb1
I20260926 11:38:17.934414 15730 raft_consensus.cc:2726] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:38:18.166642 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:18.165996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:38723)
0926 11:38:18.166423 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:18.166526 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":108,"threads_started":1}
I20260926 11:38:18.309358 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:18.308757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:55619)
0926 11:38:18.309128 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:18.309213 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":98,"threads_started":1}
I20260926 11:38:18.513595 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:18.512929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:37717)
0926 11:38:18.513395 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:18.513467 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":165,"threads_started":1}
I20260926 11:38:19.168766 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:19.168009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:53799)
0926 11:38:19.168478 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:19.168579 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":90,"threads_started":1}
I20260926 11:38:19.310999 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:19.310478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:58515)
0926 11:38:19.310804 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:19.310876 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
I20260926 11:38:19.515105 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:19.514653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:45393)
0926 11:38:19.514946 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:19.514996 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":112,"threads_started":1}
I20260926 11:38:19.912710 15737 raft_consensus.cc:482] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:19.912875 15737 raft_consensus.cc:504] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:19.913466 15737 leader_election.cc:288] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307), 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639)
W20260926 11:38:19.914454 15334 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
W20260926 11:38:19.914682 15335 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
I20260926 11:38:19.914788 15335 leader_election.cc:302] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd0f2c6e00c24e52b7b1078dc836a17a; no voters: 3b6557ae601448db8938555056bb09dd, 973fe1bffa2f428a936f2b9215650bb1
I20260926 11:38:19.914942 15737 raft_consensus.cc:2726] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:38:20.170745 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:20.170082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:42179)
0926 11:38:20.170537 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:20.170622 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":156,"threads_started":1}
I20260926 11:38:20.313092 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:20.312421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:56531)
0926 11:38:20.312826 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:20.312919 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":88,"threads_started":1}
I20260926 11:38:20.516942 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:20.516313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:33297)
0926 11:38:20.516718 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:20.516835 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
I20260926 11:38:21.172478 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:21.171916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:49749)
0926 11:38:21.172290 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:21.172370 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":101,"threads_started":1}
I20260926 11:38:21.315208 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:21.314431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:42837)
0926 11:38:21.314971 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:21.315074 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":124,"threads_started":1}
I20260926 11:38:21.518744 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:21.518060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:33861)
0926 11:38:21.518499 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:21.518598 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":94,"threads_started":1}
I20260926 11:38:22.173949 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:22.173499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:37297)
0926 11:38:22.173774 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:22.173845 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":111,"threads_started":1}
I20260926 11:38:22.317144 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:22.316499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:45781)
0926 11:38:22.316895 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:22.317010 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":106,"threads_started":1}
I20260926 11:38:22.520648 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:22.520123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:35481)
0926 11:38:22.520459 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:22.520522 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":136,"threads_started":1}
I20260926 11:38:23.175691 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:23.174944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:33767)
0926 11:38:23.175360 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:23.175459 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":85,"threads_started":1}
I20260926 11:38:23.318895 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:23.318288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:60219)
0926 11:38:23.318699 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:23.318780 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":125,"threads_started":1}
I20260926 11:38:23.522416 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:23.521837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:45439)
0926 11:38:23.522216 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:23.522279 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":116,"threads_started":1}
I20260926 11:38:24.177451 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:24.176978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:55879)
0926 11:38:24.177286 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:24.177351 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260926 11:38:24.320457 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:24.319947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:35991)
0926 11:38:24.320273 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:24.320353 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
I20260926 11:38:24.524173 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:24.523682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:38831)
0926 11:38:24.523992 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:24.524058 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260926 11:38:24.819978 15753 raft_consensus.cc:482] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:24.820199 15753 raft_consensus.cc:504] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:24.820644 15753 leader_election.cc:288] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307), 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639)
W20260926 11:38:24.821441 15335 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
W20260926 11:38:24.821578 15334 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
I20260926 11:38:24.821681 15334 leader_election.cc:302] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd0f2c6e00c24e52b7b1078dc836a17a; no voters: 3b6557ae601448db8938555056bb09dd, 973fe1bffa2f428a936f2b9215650bb1
I20260926 11:38:24.821861 15753 raft_consensus.cc:2726] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:38:25.178813 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:25.178366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:47311)
0926 11:38:25.178650 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:25.178714 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260926 11:38:25.321815 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:25.321332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:41611)
0926 11:38:25.321665 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:25.321719 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":89,"threads_started":1}
I20260926 11:38:25.525653 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:25.525192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:58835)
0926 11:38:25.525490 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:25.525555 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":101,"threads_started":1}
I20260926 11:38:26.180433 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:26.179947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:51777)
0926 11:38:26.180254 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:26.180324 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
I20260926 11:38:26.323514 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:26.322836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:39915)
0926 11:38:26.323232 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:26.323361 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260926 11:38:26.527173 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:26.526647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:53831)
0926 11:38:26.526971 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:26.527041 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260926 11:38:27.182075 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:27.181484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:40055)
0926 11:38:27.181863 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:27.181944 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260926 11:38:27.325189 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:27.324688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:36675)
0926 11:38:27.324989 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:27.325054 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260926 11:38:27.528848 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:27.528288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:52375)
0926 11:38:27.528679 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:27.528745 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":90,"threads_started":1}
I20260926 11:38:28.183503 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:28.183025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:47107)
0926 11:38:28.183321 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:28.183385 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
I20260926 11:38:28.326984 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:28.326334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:45885)
0926 11:38:28.326751 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:28.326835 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":130,"threads_started":1}
I20260926 11:38:28.530591 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:28.529977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.3:39021)
0926 11:38:28.530373 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:28.530456 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260926 11:38:28.918572 15766 raft_consensus.cc:482] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:28.918777 15766 raft_consensus.cc:504] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973fe1bffa2f428a936f2b9215650bb1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45307 } } peers { permanent_uuid: "3b6557ae601448db8938555056bb09dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41639 } } peers { permanent_uuid: "dd0f2c6e00c24e52b7b1078dc836a17a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34453 } }
I20260926 11:38:28.919423 15766 leader_election.cc:288] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307), 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639)
W20260926 11:38:28.920384 15334 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b6557ae601448db8938555056bb09dd (127.0.98.3:41639): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
W20260926 11:38:28.920490 15335 leader_election.cc:341] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 973fe1bffa2f428a936f2b9215650bb1 (127.0.98.2:45307): Not found: Tablet not found: 24d008b2da2d4ae888e8ff660e65f733
I20260926 11:38:28.920542 15335 leader_election.cc:302] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd0f2c6e00c24e52b7b1078dc836a17a; no voters: 3b6557ae601448db8938555056bb09dd, 973fe1bffa2f428a936f2b9215650bb1
I20260926 11:38:28.920707 15766 raft_consensus.cc:2726] T 24d008b2da2d4ae888e8ff660e65f733 P dd0f2c6e00c24e52b7b1078dc836a17a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:38:29.185432 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:29.184876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.1:51545)
0926 11:38:29.185216 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:29.185293 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260926 11:38:29.328794 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:29.328296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40241 (local address 127.0.98.2:52351)
0926 11:38:29.328597 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:29.328689 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
I20260926 11:38:29.505309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 15282
I20260926 11:38:29.511428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 15448
I20260926 11:38:29.517660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 15584
2026-09-26T11:38:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19503 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:38:29 dist-test-slave-4pw5 krb5kdc[15771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:38:29 dist-test-slave-4pw5 krb5kdc[15771](info): set up 2 sockets
Sep 26 11:38:29 dist-test-slave-4pw5 krb5kdc[15771](info): commencing operation
krb5kdc: starting...
W20260926 11:38:30.604333 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:38:30 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:38:30.744472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40579
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:30.868462 15784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:30.868844 15784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:30.868928 15784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:30.874897 15784 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 11:38:30.874991 15784 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 11:38:30.875021 15784 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 11:38:30.875046 15784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:30.875075 15784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:30.875118 15784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:30.875156 15784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:30.875190 15784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:30.883627 15784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:30.885179 15784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:30.886650 15784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:30.898641 15792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:30.898890 15789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:30.898715 15790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:30.901497 15784 server_base.cc:1034] running on GCE node
I20260926 11:38:30.902324 15784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:30.903796 15784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:30.905068 15784 hybrid_clock.cc:648] HybridClock initialized: now 1790422710905047 us; error 47 us; skew 500 ppm
Sep 26 11:38:30 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:30.908844 15784 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:38:30.910219 15784 webserver.cc:460] Webserver started at http://127.0.98.62:32895/ using document root <none> and password file <none>
I20260926 11:38:30.911060 15784 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:30.911165 15784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:30.911428 15784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:30.914594 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "070eca456a2f4757a11f2e36a093dda9"
format_stamp: "Formatted at 2026-09-26 11:38:30 on dist-test-slave-4pw5"
server_key: "82502e8c2c2482320f2dd37e7f6cc678"
server_key_iv: "751921f512c14f34b146f3f413047741"
server_key_version: "encryptionkey@0"
I20260926 11:38:30.915347 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "070eca456a2f4757a11f2e36a093dda9"
format_stamp: "Formatted at 2026-09-26 11:38:30 on dist-test-slave-4pw5"
server_key: "82502e8c2c2482320f2dd37e7f6cc678"
server_key_iv: "751921f512c14f34b146f3f413047741"
server_key_version: "encryptionkey@0"
I20260926 11:38:30.920266 15784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 11:38:30.923277 15799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:30.924921 15784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:30.925127 15784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "070eca456a2f4757a11f2e36a093dda9"
format_stamp: "Formatted at 2026-09-26 11:38:30 on dist-test-slave-4pw5"
server_key: "82502e8c2c2482320f2dd37e7f6cc678"
server_key_iv: "751921f512c14f34b146f3f413047741"
server_key_version: "encryptionkey@0"
I20260926 11:38:30.925290 15784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:30.960569 15784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:30.964787 15784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:30.965042 15784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:30.973526 15784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40579
I20260926 11:38:30.973896 15851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40579 every 8 connection(s)
I20260926 11:38:30.974928 15784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:38:30.978283 15852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:30.983099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 15784
I20260926 11:38:30.983248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:38:30.983708 392 external_mini_cluster.cc:1428] Setting key a87a04a6060ea8182507f9545546ec52
I20260926 11:38:30.985086 15852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Bootstrap starting.
I20260926 11:38:30.987841 15852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:30.988853 15852 log.cc:824] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:30.991225 15852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: No bootstrap required, opened a new log
Sep 26 11:38:30 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:30.996061 15852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:38:30.996373 15852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:30.996457 15852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070eca456a2f4757a11f2e36a093dda9, State: Initialized, Role: FOLLOWER
I20260926 11:38:30.997013 15852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:38:30.997184 15852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:38:30.997277 15852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:38:30.997394 15852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:30.998678 15852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:38:30.999105 15852 leader_election.cc:302] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070eca456a2f4757a11f2e36a093dda9; no voters:
I20260926 11:38:30.999521 15852 leader_election.cc:288] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:38:30.999727 15857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:38:30.999969 15857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 1 LEADER]: Becoming Leader. State: Replica: 070eca456a2f4757a11f2e36a093dda9, State: Running, Role: LEADER
I20260926 11:38:31.000399 15857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:38:31.000977 15852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:38:31.001209 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:30.985446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.0.98.62:40579)
0926 11:38:30.985939 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:30.985961 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:30.985999 (+ 38us) server_negotiation.cc:407] Connection header received
0926 11:38:30.986746 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:30.986764 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:30.987132 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:30.987449 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:30.988728 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:30.989813 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:30.990984 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:30.991749 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:30.991896 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:30.994909 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:30.994916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:30.994933 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:30.994968 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:30.997216 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:30.997867 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:30.997873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:30.997878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:30.998042 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:30.998395 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:30.998398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:30.998399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:30.999022 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:30.999189 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:30.999667 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.000071 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":165,"threads_started":1}
I20260926 11:38:31.003018 15859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070eca456a2f4757a11f2e36a093dda9. Latest consensus state: current_term: 1 leader_uuid: "070eca456a2f4757a11f2e36a093dda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } } }
I20260926 11:38:31.003400 15858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070eca456a2f4757a11f2e36a093dda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } } }
I20260926 11:38:31.004128 15858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:31.003925 15859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: This master's current role is: LEADER
W20260926 11:38:31.006915 15871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:38:31.006994 15871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:38:31.007081 15873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:38:31.007977 15873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:38:31.013863 15873 catalog_manager.cc:1347] Generated new cluster ID: e5731ba5edfe4066ab70091a01bcdfd4
I20260926 11:38:31.013967 15873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:38:31.020844 15873 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:38:31.021907 15873 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:38:31.033847 15873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Generated new TSK 0
I20260926 11:38:31.034688 15873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:38:31.075405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40579
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:31.201305 15878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:31.201730 15878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:31.201838 15878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:31.206823 15878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:31.206902 15878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:31.206969 15878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:38:31.212914 15878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:31.214112 15878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:31.215190 15878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:31.226192 15883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:31.226192 15884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:31.226199 15886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:31.226747 15878 server_base.cc:1034] running on GCE node
I20260926 11:38:31.227248 15878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:31.228055 15878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:31.229245 15878 hybrid_clock.cc:648] HybridClock initialized: now 1790422711229224 us; error 35 us; skew 500 ppm
Sep 26 11:38:31 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:31.237435 15878 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:38:31.238615 15878 webserver.cc:460] Webserver started at http://127.0.98.1:42239/ using document root <none> and password file <none>
I20260926 11:38:31.239193 15878 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:31.239256 15878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:31.239431 15878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:31.241809 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "62534bea9f904d07a2d86c334d169a34"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "9e8d46ba7f9c71dc74d7ba89267581b7"
server_key_iv: "460373a4922ece0b62a06e70846f069e"
server_key_version: "encryptionkey@0"
I20260926 11:38:31.242381 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "62534bea9f904d07a2d86c334d169a34"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "9e8d46ba7f9c71dc74d7ba89267581b7"
server_key_iv: "460373a4922ece0b62a06e70846f069e"
server_key_version: "encryptionkey@0"
I20260926 11:38:31.246383 15878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 11:38:31.249397 15893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:31.251012 15878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:38:31.251199 15878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "62534bea9f904d07a2d86c334d169a34"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "9e8d46ba7f9c71dc74d7ba89267581b7"
server_key_iv: "460373a4922ece0b62a06e70846f069e"
server_key_version: "encryptionkey@0"
I20260926 11:38:31.251351 15878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:31.293840 15878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:31.297536 15878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:31.297772 15878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:31.298435 15878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:31.299350 15878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:31.299422 15878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:31.299497 15878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:31.299552 15878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:31.310808 15878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44635
I20260926 11:38:31.310871 16006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44635 every 8 connection(s)
I20260926 11:38:31.312000 15878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:38:31.314796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 15878
I20260926 11:38:31.314975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:38:31.315315 392 external_mini_cluster.cc:1428] Setting key b4a76c9055b65bf65efd90a30c5fab9d
Sep 26 11:38:31 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:38:31.329023 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.314279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47655 (local address 127.0.98.62:40579)
0926 11:38:31.314433 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.314437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.315664 (+ 1227us) server_negotiation.cc:407] Connection header received
0926 11:38:31.317327 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.317330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.317390 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.317490 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:31.318868 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.319480 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.320845 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.321115 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.321249 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.324501 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:31.324504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:31.324507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:31.324537 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:31.326437 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:31.327168 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:31.327176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:31.327177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:31.327260 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:31.327661 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:31.327664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:31.327666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:31.327886 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:31.328003 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.328675 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.328838 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:38:31.329931 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.314737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:47655)
0926 11:38:31.315413 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:31.315486 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:31.317094 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:31.317714 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:31.317722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:31.318146 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:31.318700 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.318715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.319665 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.319671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.320692 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.320703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.321303 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.321308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.321514 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.322266 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:31.322285 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:31.324255 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:31.326596 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:31.326603 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:31.326632 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:31.327036 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:31.327359 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:31.327364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:31.327366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:31.327530 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:31.328003 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:31.328012 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:31.328405 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 11:38:31.328616 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 11:38:31.328997 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":195,"threads_started":1}
I20260926 11:38:31.331415 16007 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40579
I20260926 11:38:31.331779 16007 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:31.332412 16007 heartbeater.cc:511] Master 127.0.98.62:40579 requested a full tablet report, sending...
I20260926 11:38:31.334390 15816 ts_manager.cc:194] Registered new tserver with Master: 62534bea9f904d07a2d86c334d169a34 (127.0.98.1:44635)
I20260926 11:38:31.335924 15816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:38:31.347674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40579
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:31.470770 16012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:31.471098 16012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:31.471146 16012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:31.475462 16012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:31.475540 16012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:31.475636 16012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:38:31.481426 16012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:31.482586 16012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:31.483646 16012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:31.494457 16017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:31.494660 16018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:31.494765 16020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:31.494875 16012 server_base.cc:1034] running on GCE node
I20260926 11:38:31.495334 16012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:31.496131 16012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:31.497332 16012 hybrid_clock.cc:648] HybridClock initialized: now 1790422711497313 us; error 32 us; skew 500 ppm
Sep 26 11:38:31 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:31.500552 16012 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:38:31.501642 16012 webserver.cc:460] Webserver started at http://127.0.98.2:39181/ using document root <none> and password file <none>
I20260926 11:38:31.502355 16012 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:31.502436 16012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:31.502661 16012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:31.505007 16012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "5abe887d28ef40cc8c41d1673b7a5b7b"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "7b9c244b291d51669be3de4977840fea"
server_key_iv: "37e98dd1cfbea95d81cf1187a18f0c1a"
server_key_version: "encryptionkey@0"
I20260926 11:38:31.505628 16012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5abe887d28ef40cc8c41d1673b7a5b7b"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "7b9c244b291d51669be3de4977840fea"
server_key_iv: "37e98dd1cfbea95d81cf1187a18f0c1a"
server_key_version: "encryptionkey@0"
I20260926 11:38:31.509730 16012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 11:38:31.512652 16027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:31.514186 16012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:31.514346 16012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "5abe887d28ef40cc8c41d1673b7a5b7b"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "7b9c244b291d51669be3de4977840fea"
server_key_iv: "37e98dd1cfbea95d81cf1187a18f0c1a"
server_key_version: "encryptionkey@0"
I20260926 11:38:31.514482 16012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:31.527458 16012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:31.531976 16012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:31.532224 16012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:31.532881 16012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:31.533836 16012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:31.533892 16012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:31.533962 16012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:31.534000 16012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:31.543623 16012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38317
I20260926 11:38:31.543675 16140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38317 every 8 connection(s)
I20260926 11:38:31.544646 16012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:38:31.545374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 16012
I20260926 11:38:31.545493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:38:31.545804 392 external_mini_cluster.cc:1428] Setting key 51b60e6103377b4cb1c9f4635dae25c0
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 26 11:38:31 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:31.565217 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.546864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43267 (local address 127.0.98.62:40579)
0926 11:38:31.547020 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.547025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.548220 (+ 1195us) server_negotiation.cc:407] Connection header received
0926 11:38:31.549750 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.549754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.549823 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.549951 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:31.551384 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.552162 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.553484 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.553719 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.553836 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.559999 (+ 6163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:31.560007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:31.560014 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:31.560057 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:31.562059 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:31.563015 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:31.563020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:31.563023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:31.563146 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:31.563703 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:31.563706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:31.563707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:31.563926 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:31.564021 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.564894 (+ 873us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.565057 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 11:38:31.566488 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.547372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:43267)
0926 11:38:31.548008 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:31.548056 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:31.549488 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:31.550186 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:31.550194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:31.550635 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:31.551198 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.551214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.552335 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.552338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.553335 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.553345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.553887 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.553891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.554075 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.554774 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:31.554792 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:31.559744 (+ 4952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:31.562252 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:31.562263 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:31.562290 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:31.562786 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:31.563259 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:31.563265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:31.563271 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:31.563523 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:31.564060 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:31.564070 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:31.564516 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 11:38:31.564797 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 11:38:31.565270 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":168,"threads_started":1}
I20260926 11:38:31.568064 16141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40579
I20260926 11:38:31.568387 16141 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:31.569072 16141 heartbeater.cc:511] Master 127.0.98.62:40579 requested a full tablet report, sending...
I20260926 11:38:31.570578 15816 ts_manager.cc:194] Registered new tserver with Master: 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317)
I20260926 11:38:31.571293 15816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:38:31.580966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40579
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:31.701025 16146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:31.701340 16146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:31.701434 16146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:31.708144 16146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:31.708271 16146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:31.708398 16146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:38:31.714793 16146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:31.716372 16146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:31.717592 16146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:31.728266 16151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:31.728392 16154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:31.728531 16146 server_base.cc:1034] running on GCE node
W20260926 11:38:31.728266 16152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:31.729182 16146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:31.730042 16146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:31.731259 16146 hybrid_clock.cc:648] HybridClock initialized: now 1790422711731233 us; error 37 us; skew 500 ppm
Sep 26 11:38:31 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:31.734714 16146 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:38:31.735987 16146 webserver.cc:460] Webserver started at http://127.0.98.3:44385/ using document root <none> and password file <none>
I20260926 11:38:31.736681 16146 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:31.736785 16146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:31.737041 16146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:38:31.739331 16146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "773102a5cfae4965949f7eb6695fc8e5"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "2243f7e08a30b79264e90591ed4cf0bc"
server_key_iv: "d84a2c64899012a345e7a6df83c40bcc"
server_key_version: "encryptionkey@0"
I20260926 11:38:31.740046 16146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "773102a5cfae4965949f7eb6695fc8e5"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "2243f7e08a30b79264e90591ed4cf0bc"
server_key_iv: "d84a2c64899012a345e7a6df83c40bcc"
server_key_version: "encryptionkey@0"
I20260926 11:38:31.744874 16146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 11:38:31.748229 16161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:31.749965 16146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:38:31.750315 16146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "773102a5cfae4965949f7eb6695fc8e5"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "2243f7e08a30b79264e90591ed4cf0bc"
server_key_iv: "d84a2c64899012a345e7a6df83c40bcc"
server_key_version: "encryptionkey@0"
I20260926 11:38:31.750496 16146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:31.773905 16146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:31.777805 16146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:31.778057 16146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:31.778791 16146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:31.779837 16146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:31.779906 16146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:31.779978 16146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:31.780018 16146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:31.792824 16146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41131
I20260926 11:38:31.792863 16274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41131 every 8 connection(s)
I20260926 11:38:31.793900 16146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:38:31.799034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 16146
I20260926 11:38:31.799157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:38:31.799746 392 external_mini_cluster.cc:1428] Setting key 0869ddcaa01a9db84ec32fbbc766da96
Sep 26 11:38:31 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:31.809628 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.796205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53961 (local address 127.0.98.62:40579)
0926 11:38:31.796432 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.796437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.797286 (+ 849us) server_negotiation.cc:407] Connection header received
0926 11:38:31.798641 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.798645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.798707 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.798822 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:31.800329 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.800883 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.802239 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.802467 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.802583 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.805678 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:31.805682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:31.805684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:31.805713 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:31.807163 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:31.807857 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:31.807866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:31.807868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:31.807952 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:31.808339 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:31.808343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:31.808344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:31.808551 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:31.808661 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.809261 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.809439 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 11:38:31.810760 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.796602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:53961)
0926 11:38:31.797115 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:31.797162 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:31.798402 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:31.799028 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:31.799036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:31.799538 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:31.800169 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.800185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.801024 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.801029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.802085 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.802098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.802628 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.802632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.802854 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.803509 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:31.803527 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:31.805378 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:31.807321 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:31.807329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:31.807348 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:31.807742 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:31.808062 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:31.808065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:31.808067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:31.808231 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:31.808673 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:31.808682 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:31.808982 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 11:38:31.809202 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 11:38:31.809588 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":113,"threads_started":1}
I20260926 11:38:31.812429 16275 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40579
I20260926 11:38:31.812741 16275 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:31.813346 16275 heartbeater.cc:511] Master 127.0.98.62:40579 requested a full tablet report, sending...
I20260926 11:38:31.814503 15815 ts_manager.cc:194] Registered new tserver with Master: 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131)
I20260926 11:38:31.815158 15815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53961
I20260926 11:38:31.827067 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:38:31.838301 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.829382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.0.98.62:40579)
0926 11:38:31.829527 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.829532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.829569 (+ 37us) server_negotiation.cc:407] Connection header received
0926 11:38:31.829798 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.829801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.829850 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.829928 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:31.831006 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.831715 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.832676 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.832951 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.833074 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.834720 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:31.834723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:31.834726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:31.834755 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:31.836437 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:31.836975 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:31.836979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:31.836981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:31.837068 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:31.837345 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:31.837349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:31.837351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:31.837603 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:31.837743 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.837993 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.838116 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:38:31.841974 15815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:38:31.845709 15815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:38:31.863014 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.857565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44635 (local address 127.0.0.1:33864)
0926 11:38:31.857906 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:31.857918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:31.857988 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:31.858837 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:31.858839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:31.858860 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:31.858947 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.858952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.860463 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.860467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.861347 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.861357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.862392 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.862396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.862524 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.862569 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 11:38:31.862704 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 11:38:31.862851 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":12,"thread_start_us":111,"threads_started":1}
I20260926 11:38:31.863255 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.856280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41131 (local address 127.0.0.1:47592)
0926 11:38:31.856926 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:31.856984 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:31.857122 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:31.858199 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:31.858205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:31.858242 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:31.858625 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.858632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.860298 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.860301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.861350 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.861361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.862823 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.862827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.862922 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.862945 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:38:31.863014 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 11:38:31.863085 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":45,"spinlock_wait_cycles":5248,"thread_start_us":97,"threads_started":1}
I20260926 11:38:31.863775 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.856497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.0.98.3:41131)
0926 11:38:31.857097 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.857112 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.857136 (+ 24us) server_negotiation.cc:407] Connection header received
0926 11:38:31.857351 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.857357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.858007 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.858158 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:31.858766 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.860138 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.861675 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.862520 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.862620 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.862758 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.863471 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.863627 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":181,"threads_started":1}
I20260926 11:38:31.863024 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.857894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.0.98.1:44635)
0926 11:38:31.858335 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.858343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.858367 (+ 24us) server_negotiation.cc:407] Connection header received
0926 11:38:31.858445 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.858452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.858627 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.858836 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:31.859068 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.860371 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.861620 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.862246 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.862391 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.862625 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.862753 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.862862 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":83,"threads_started":1}
I20260926 11:38:31.864138 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.856397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38317 (local address 127.0.0.1:59968)
0926 11:38:31.857574 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:31.857588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:31.857712 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:31.858212 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:31.858215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:31.858244 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:31.858433 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.858440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.859853 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.859856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.862303 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.862315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.863489 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.863679 (+ 190us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.863789 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.863827 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 11:38:31.863882 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 11:38:31.863969 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":1094,"spinlock_wait_cycles":8064,"thread_start_us":91,"threads_started":1}
I20260926 11:38:31.865644 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.857138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.0.98.2:38317)
0926 11:38:31.857545 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.857556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.857700 (+ 144us) server_negotiation.cc:407] Connection header received
0926 11:38:31.857844 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.857849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.858044 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.858352 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:31.858542 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.859515 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.862468 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.863350 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.865098 (+ 1748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.865285 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.865343 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.865442 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":96,"threads_started":1}
I20260926 11:38:31.867025 16209 tablet_service.cc:1467] Processing CreateTablet for tablet aa71ae87047e445ab45b89eb62a16a14 (DEFAULT_TABLE table=test-table [id=49ca012c34874207922f8eb1c43f8813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:31.867504 15941 tablet_service.cc:1467] Processing CreateTablet for tablet aa71ae87047e445ab45b89eb62a16a14 (DEFAULT_TABLE table=test-table [id=49ca012c34874207922f8eb1c43f8813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:31.868625 15941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa71ae87047e445ab45b89eb62a16a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:31.868368 16209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa71ae87047e445ab45b89eb62a16a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:31.869145 16075 tablet_service.cc:1467] Processing CreateTablet for tablet aa71ae87047e445ab45b89eb62a16a14 (DEFAULT_TABLE table=test-table [id=49ca012c34874207922f8eb1c43f8813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:31.870198 16075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa71ae87047e445ab45b89eb62a16a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:31.876641 16292 tablet_bootstrap.cc:492] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: Bootstrap starting.
I20260926 11:38:31.878293 16294 tablet_bootstrap.cc:492] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: Bootstrap starting.
I20260926 11:38:31.878962 16293 tablet_bootstrap.cc:492] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Bootstrap starting.
I20260926 11:38:31.880841 16292 tablet_bootstrap.cc:654] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:31.881224 16294 tablet_bootstrap.cc:654] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:31.881253 16293 tablet_bootstrap.cc:654] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:31.881880 16292 log.cc:824] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:31.882121 16293 log.cc:824] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:31.882121 16294 log.cc:824] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:31.884383 16294 tablet_bootstrap.cc:492] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: No bootstrap required, opened a new log
I20260926 11:38:31.884445 16293 tablet_bootstrap.cc:492] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: No bootstrap required, opened a new log
I20260926 11:38:31.884445 16292 tablet_bootstrap.cc:492] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: No bootstrap required, opened a new log
I20260926 11:38:31.884644 16294 ts_tablet_manager.cc:1397] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 11:38:31.884644 16293 ts_tablet_manager.cc:1397] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:38:31.884644 16292 ts_tablet_manager.cc:1397] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 11:38:31.888469 16294 raft_consensus.cc:348] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:31.888793 16294 raft_consensus.cc:374] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:31.888849 16294 raft_consensus.cc:729] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abe887d28ef40cc8c41d1673b7a5b7b, State: Initialized, Role: FOLLOWER
I20260926 11:38:31.888850 16292 raft_consensus.cc:348] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:31.889142 16292 raft_consensus.cc:374] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:31.889201 16292 raft_consensus.cc:729] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773102a5cfae4965949f7eb6695fc8e5, State: Initialized, Role: FOLLOWER
I20260926 11:38:31.889307 16293 raft_consensus.cc:348] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:31.889418 16294 consensus_queue.cc:260] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:31.889551 16293 raft_consensus.cc:374] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:31.889603 16293 raft_consensus.cc:729] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62534bea9f904d07a2d86c334d169a34, State: Initialized, Role: FOLLOWER
I20260926 11:38:31.889634 16292 consensus_queue.cc:260] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:31.890057 16293 consensus_queue.cc:260] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:31.890236 16141 heartbeater.cc:503] Master 127.0.98.62:40579 was elected leader, sending a full tablet report...
I20260926 11:38:31.890654 16294 ts_tablet_manager.cc:1428] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:38:31.890673 16292 ts_tablet_manager.cc:1428] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:38:31.891004 16275 heartbeater.cc:503] Master 127.0.98.62:40579 was elected leader, sending a full tablet report...
I20260926 11:38:31.891211 16293 ts_tablet_manager.cc:1428] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:38:31.891763 16007 heartbeater.cc:503] Master 127.0.98.62:40579 was elected leader, sending a full tablet report...
I20260926 11:38:31.918090 16299 raft_consensus.cc:482] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:31.918383 16299 raft_consensus.cc:504] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:31.919736 16299 leader_election.cc:288] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317), 62534bea9f904d07a2d86c334d169a34 (127.0.98.1:44635)
I20260926 11:38:31.924094 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.920003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44635 (local address 127.0.98.3:36687)
0926 11:38:31.920216 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:31.920233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:31.920359 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:31.920739 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:31.920741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:31.920754 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:31.920898 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.920905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.921742 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.921747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.922799 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.922811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.923603 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.923608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.923721 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.923766 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 11:38:31.923859 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 11:38:31.923941 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":129}
I20260926 11:38:31.924094 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.920318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36687 (local address 127.0.98.1:44635)
0926 11:38:31.920446 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.920452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.920469 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:38:31.920542 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.920545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.920601 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.920674 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:31.921005 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.921612 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.922964 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.923428 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.923575 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.923664 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.923876 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.923957 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:38:31.924166 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.920129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42183 (local address 127.0.98.2:38317)
0926 11:38:31.920274 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.920279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.920511 (+ 232us) server_negotiation.cc:407] Connection header received
0926 11:38:31.920718 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.920723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.920778 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.920856 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:31.921219 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.921729 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.923022 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.923466 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.923605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.923672 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.923929 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.924022 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:38:31.924731 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.920023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38317 (local address 127.0.98.3:42183)
0926 11:38:31.920429 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:31.920444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:31.920593 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:31.920886 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:31.920889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:31.920907 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:31.921039 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.921049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.921846 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.921852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.922841 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:31.922852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.923617 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.923621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:31.923794 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.923813 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 11:38:31.923910 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 11:38:31.924583 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":40,"thread_start_us":170,"threads_started":1}
I20260926 11:38:31.924993 15961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71ae87047e445ab45b89eb62a16a14" candidate_uuid: "773102a5cfae4965949f7eb6695fc8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62534bea9f904d07a2d86c334d169a34" is_pre_election: true
I20260926 11:38:31.925361 15961 raft_consensus.cc:2454] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773102a5cfae4965949f7eb6695fc8e5 in term 0.
I20260926 11:38:31.925516 16095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71ae87047e445ab45b89eb62a16a14" candidate_uuid: "773102a5cfae4965949f7eb6695fc8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" is_pre_election: true
I20260926 11:38:31.925832 16095 raft_consensus.cc:2454] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773102a5cfae4965949f7eb6695fc8e5 in term 0.
I20260926 11:38:31.925933 16164 leader_election.cc:302] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62534bea9f904d07a2d86c334d169a34, 773102a5cfae4965949f7eb6695fc8e5; no voters:
I20260926 11:38:31.926297 16299 raft_consensus.cc:2781] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:38:31.926412 16299 raft_consensus.cc:482] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:38:31.926460 16299 raft_consensus.cc:3037] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:31.927996 16299 raft_consensus.cc:504] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:31.928408 16299 leader_election.cc:288] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [CANDIDATE]: Term 1 election: Requested vote from peers 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317), 62534bea9f904d07a2d86c334d169a34 (127.0.98.1:44635)
I20260926 11:38:31.928953 15961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71ae87047e445ab45b89eb62a16a14" candidate_uuid: "773102a5cfae4965949f7eb6695fc8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62534bea9f904d07a2d86c334d169a34"
I20260926 11:38:31.929155 15961 raft_consensus.cc:3037] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:31.930469 15961 raft_consensus.cc:2454] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773102a5cfae4965949f7eb6695fc8e5 in term 1.
I20260926 11:38:31.930897 16095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71ae87047e445ab45b89eb62a16a14" candidate_uuid: "773102a5cfae4965949f7eb6695fc8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b"
I20260926 11:38:31.930968 16164 leader_election.cc:302] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62534bea9f904d07a2d86c334d169a34, 773102a5cfae4965949f7eb6695fc8e5; no voters:
I20260926 11:38:31.931067 16095 raft_consensus.cc:3037] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:31.931195 16299 raft_consensus.cc:2781] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:38:31.931510 16299 raft_consensus.cc:686] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 1 LEADER]: Becoming Leader. State: Replica: 773102a5cfae4965949f7eb6695fc8e5, State: Running, Role: LEADER
I20260926 11:38:31.932288 16299 consensus_queue.cc:237] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:31.932524 16095 raft_consensus.cc:2454] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773102a5cfae4965949f7eb6695fc8e5 in term 1.
I20260926 11:38:31.936142 15814 catalog_manager.cc:5495] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "773102a5cfae4965949f7eb6695fc8e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } health_report { overall_health: UNKNOWN } } }
I20260926 11:38:31.965052 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:31.961303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.0.98.3:41131)
0926 11:38:31.961429 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:31.961432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:31.961486 (+ 54us) server_negotiation.cc:407] Connection header received
0926 11:38:31.961564 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:31.961567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:31.961624 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:31.961710 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:38:31.961915 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.962413 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.963344 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:31.963702 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:31.963807 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:31.964142 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:38:31.964613 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:38:31.964717 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:31.964832 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 11:38:31.964890 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 11:38:31.971856 15961 raft_consensus.cc:1261] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Refusing update from remote peer 773102a5cfae4965949f7eb6695fc8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:31.971856 16095 raft_consensus.cc:1261] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [term 1 FOLLOWER]: Refusing update from remote peer 773102a5cfae4965949f7eb6695fc8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:31.972641 16299 consensus_queue.cc:1035] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:31.972880 16302 consensus_queue.cc:1035] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:31.985003 16306 mvcc.cc:205] Tried to move back new op lower bound from 7333571428228214784 to 7333571428078383104. Current Snapshot: MvccSnapshot[applied={T|T < 7333571428228214784}]
I20260926 11:38:31.992386 15814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:31.992568 15814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:31.995147 15814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Sending DeleteTablet for 3 replicas of tablet aa71ae87047e445ab45b89eb62a16a14
I20260926 11:38:31.996141 16075 tablet_service.cc:1514] Processing DeleteTablet for tablet aa71ae87047e445ab45b89eb62a16a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59968
I20260926 11:38:31.996140 16209 tablet_service.cc:1514] Processing DeleteTablet for tablet aa71ae87047e445ab45b89eb62a16a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47592
I20260926 11:38:31.996243 15941 tablet_service.cc:1514] Processing DeleteTablet for tablet aa71ae87047e445ab45b89eb62a16a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33864
I20260926 11:38:31.996685 16315 tablet_replica.cc:330] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: stopping tablet replica
I20260926 11:38:31.996937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 15878
I20260926 11:38:31.997056 16315 raft_consensus.cc:2229] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:38:31.997227 16314 tablet_replica.cc:330] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: stopping tablet replica
I20260926 11:38:31.996930 16316 tablet_replica.cc:330] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: stopping tablet replica
I20260926 11:38:31.997424 16315 raft_consensus.cc:2258] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:31.997699 16314 raft_consensus.cc:2229] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:38:31.998158 16314 raft_consensus.cc:2258] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:31.999189 16315 ts_tablet_manager.cc:1905] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:31.999688 16314 ts_tablet_manager.cc:1905] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:32.002399 16315 ts_tablet_manager.cc:1918] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:32.002532 16315 log.cc:1196] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/aa71ae87047e445ab45b89eb62a16a14
I20260926 11:38:32.002717 16314 ts_tablet_manager.cc:1918] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:32.002875 16315 ts_tablet_manager.cc:1939] T aa71ae87047e445ab45b89eb62a16a14 P 5abe887d28ef40cc8c41d1673b7a5b7b: Deleting consensus metadata
I20260926 11:38:32.002916 16314 log.cc:1196] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/aa71ae87047e445ab45b89eb62a16a14
I20260926 11:38:32.003345 16314 ts_tablet_manager.cc:1939] T aa71ae87047e445ab45b89eb62a16a14 P 773102a5cfae4965949f7eb6695fc8e5: Deleting consensus metadata
I20260926 11:38:32.004452 15800 catalog_manager.cc:4841] TS 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317): tablet aa71ae87047e445ab45b89eb62a16a14 (table test-table [id=49ca012c34874207922f8eb1c43f8813]) successfully deleted
I20260926 11:38:32.004452 15801 catalog_manager.cc:4841] TS 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131): tablet aa71ae87047e445ab45b89eb62a16a14 (table test-table [id=49ca012c34874207922f8eb1c43f8813]) successfully deleted
W20260926 11:38:32.007115 15802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44635 (error 108)
I20260926 11:38:32.008318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 16012
I20260926 11:38:32.008641 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:32.008314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44635 (local address 127.0.0.1:33868)
0926 11:38:32.008454 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:32.008535 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 11:38:32.009045 15802 catalog_manager.cc:4584] TS 62534bea9f904d07a2d86c334d169a34 (127.0.98.1:44635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa71ae87047e445ab45b89eb62a16a14: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44635: connect: Connection refused (error 111)
I20260926 11:38:32.014799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 16146
I20260926 11:38:32.021090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 15784
I20260926 11:38:52.028381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40579
--webserver_interface=127.0.98.62
--webserver_port=32895
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:38:52.150998 16318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:52.151458 16318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:52.151548 16318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:52.156366 16318 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 11:38:52.156466 16318 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 11:38:52.156482 16318 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 11:38:52.156497 16318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:38:52.156519 16318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:52.156539 16318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:52.156555 16318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:38:52.156572 16318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:38:52.162676 16318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:52.164119 16318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:52.165280 16318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:52.181743 16324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:52.181874 16327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:52.182029 16318 server_base.cc:1034] running on GCE node
W20260926 11:38:52.181743 16325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:52.183147 16318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:52.184742 16318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:52.186007 16318 hybrid_clock.cc:648] HybridClock initialized: now 1790422732185981 us; error 52 us; skew 500 ppm
Sep 26 11:38:52 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:52.189904 16318 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:38:52.191138 16318 webserver.cc:460] Webserver started at http://127.0.98.62:32895/ using document root <none> and password file <none>
I20260926 11:38:52.192065 16318 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:52.192260 16318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:52.197026 16318 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:38:52.199927 16334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:52.201748 16318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:52.201977 16318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "070eca456a2f4757a11f2e36a093dda9"
format_stamp: "Formatted at 2026-09-26 11:38:30 on dist-test-slave-4pw5"
server_key: "82502e8c2c2482320f2dd37e7f6cc678"
server_key_iv: "751921f512c14f34b146f3f413047741"
server_key_version: "encryptionkey@0"
I20260926 11:38:52.202701 16318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:52.226893 16318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:52.230484 16318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:52.230724 16318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:52.241025 16318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40579
I20260926 11:38:52.241417 16386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40579 every 8 connection(s)
I20260926 11:38:52.242436 16318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:38:52.248322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 16318
I20260926 11:38:52.248978 16387 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:38:52.248962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44635
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40579
--webserver_port=42239
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:38:52.252243 16387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Bootstrap starting.
I20260926 11:38:52.273909 16387 log.cc:824] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:52.284919 16387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:38:52.285437 16387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Bootstrap complete.
I20260926 11:38:52.290757 16387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:38:52.291308 16387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070eca456a2f4757a11f2e36a093dda9, State: Initialized, Role: FOLLOWER
I20260926 11:38:52.292047 16387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:38:52.292217 16387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:38:52.292320 16387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:38:52.292462 16387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:38:52.295032 16387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:38:52.295459 16387 leader_election.cc:302] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070eca456a2f4757a11f2e36a093dda9; no voters:
I20260926 11:38:52.295910 16387 leader_election.cc:288] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 11:38:52.296054 16391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 11:38:52.296308 16391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 2 LEADER]: Becoming Leader. State: Replica: 070eca456a2f4757a11f2e36a093dda9, State: Running, Role: LEADER
I20260926 11:38:52.296670 16391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:38:52.297487 16387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:38:52.298270 16392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "070eca456a2f4757a11f2e36a093dda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } } }
I20260926 11:38:52.298502 16392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:52.299414 16399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:38:52.299695 16393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070eca456a2f4757a11f2e36a093dda9. Latest consensus state: current_term: 2 leader_uuid: "070eca456a2f4757a11f2e36a093dda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } } }
I20260926 11:38:52.299829 16393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: This master's current role is: LEADER
I20260926 11:38:52.305872 16399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:38:52.306984 16399 catalog_manager.cc:1259] Loaded cluster ID: e5731ba5edfe4066ab70091a01bcdfd4
I20260926 11:38:52.307061 16399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:38:52.309651 16399 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:38:52.310423 16399 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Loaded TSK: 0
I20260926 11:38:52.324318 16399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Generated new TSK 1
I20260926 11:38:52.324676 16399 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:38:52.383023 16389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:52.383307 16389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:52.383353 16389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:52.387699 16389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:52.387782 16389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:52.387844 16389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:38:52.393611 16389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42239
--tserver_master_addrs=127.0.98.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:52.394814 16389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:52.395972 16389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:52.409597 16417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:52.409579 16415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:52.409790 16389 server_base.cc:1034] running on GCE node
W20260926 11:38:52.409583 16414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:52.410637 16389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:52.411409 16389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:52.412789 16389 hybrid_clock.cc:648] HybridClock initialized: now 1790422732412739 us; error 92 us; skew 500 ppm
Sep 26 11:38:52 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:52.416157 16389 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:38:52.417418 16389 webserver.cc:460] Webserver started at http://127.0.98.1:42239/ using document root <none> and password file <none>
I20260926 11:38:52.418056 16389 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:52.418172 16389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:52.422926 16389 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:38:52.426154 16424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:52.428133 16389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 11:38:52.428377 16389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "62534bea9f904d07a2d86c334d169a34"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "9e8d46ba7f9c71dc74d7ba89267581b7"
server_key_iv: "460373a4922ece0b62a06e70846f069e"
server_key_version: "encryptionkey@0"
I20260926 11:38:52.429006 16389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:52.445290 16389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:52.449090 16389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:52.449326 16389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:52.449981 16389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:52.451335 16431 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:38:52.456108 16389 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:38:52.456204 16389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 11:38:52.456261 16389 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:38:52.458248 16389 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:38:52.458324 16389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 11:38:52.458606 16431 tablet_bootstrap.cc:492] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Bootstrap starting.
I20260926 11:38:52.470472 16389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44635
I20260926 11:38:52.470680 16538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44635 every 8 connection(s)
I20260926 11:38:52.471920 16389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:38:52.478410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 16389
I20260926 11:38:52.479014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38317
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40579
--webserver_port=39181
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:38:52.481083 16431 log.cc:824] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:38:52 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:52.489892 16431 tablet_bootstrap.cc:492] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:38:52.490419 16431 tablet_bootstrap.cc:492] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Bootstrap complete.
I20260926 11:38:52.492020 16431 ts_tablet_manager.cc:1397] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.014s
I20260926 11:38:52.494867 16431 raft_consensus.cc:348] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:52.495437 16431 raft_consensus.cc:729] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62534bea9f904d07a2d86c334d169a34, State: Initialized, Role: FOLLOWER
I20260926 11:38:52.496049 16431 consensus_queue.cc:260] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:52.496485 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.474970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:54623)
0926 11:38:52.475789 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:52.475870 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:52.477017 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:52.478460 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:52.478470 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:52.479244 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:52.479969 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:52.479997 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.482302 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.482307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:52.483810 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:52.483826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.485241 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.485245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:52.485465 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.486324 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:52.486345 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:52.488944 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:52.492224 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:52.492234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:52.492250 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:52.492615 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:52.493284 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:52.493287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:52.493289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:52.493466 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:52.494457 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:52.494464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:52.494795 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 11:38:52.495001 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 11:38:52.495470 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":192,"threads_started":1}
I20260926 11:38:52.497129 16431 ts_tablet_manager.cc:1428] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:38:52.497891 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.475001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54623 (local address 127.0.98.62:40579)
0926 11:38:52.475871 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:52.475895 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:52.476067 (+ 172us) server_negotiation.cc:407] Connection header received
0926 11:38:52.477546 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:52.477559 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:52.477936 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:52.478404 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:52.480200 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.482055 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.484011 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.484940 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.485169 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.489333 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:52.489342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:52.489362 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:52.489410 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:52.492030 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.492832 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.492845 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.492857 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.493135 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.493619 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.493625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.493629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.494301 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:52.494484 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:52.495203 (+ 719us) server_negotiation.cc:299] Negotiation successful
0926 11:38:52.496362 (+ 1159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":165,"threads_started":1}
I20260926 11:38:52.500975 16539 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40579
I20260926 11:38:52.501334 16539 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:52.502170 16539 heartbeater.cc:511] Master 127.0.98.62:40579 requested a full tablet report, sending...
I20260926 11:38:52.506148 16345 ts_manager.cc:194] Registered new tserver with Master: 62534bea9f904d07a2d86c334d169a34 (127.0.98.1:44635)
I20260926 11:38:52.510412 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54623
I20260926 11:38:52.514132 16539 heartbeater.cc:503] Master 127.0.98.62:40579 was elected leader, sending a full tablet report...
Sep 26 11:38:52 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:38:52.522047 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.510013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44635 (local address 127.0.0.1:37106)
0926 11:38:52.510949 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:52.510999 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:52.511165 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:52.511997 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:52.512004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:52.512423 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:52.512536 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:52.512543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.513615 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.513620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:52.514338 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:52.514349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.515044 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.515073 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:52.515204 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.515919 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:52.515932 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:52.517851 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:52.520071 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:52.520084 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:52.520091 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:52.520355 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:52.520670 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:52.520681 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:52.520683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:52.520772 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:52.521274 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:52.521281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:52.521428 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 11:38:52.521578 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 11:38:52.521855 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":177,"threads_started":1}
I20260926 11:38:52.522047 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.510117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.0.98.1:44635)
0926 11:38:52.511388 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:52.511400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:52.511451 (+ 51us) server_negotiation.cc:407] Connection header received
0926 11:38:52.511533 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:52.511538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:52.511814 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:52.511996 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:52.512683 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.513491 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.514474 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.514939 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.515057 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.518013 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:52.518025 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:52.518048 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:52.518071 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:52.519937 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.520489 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.520495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.520496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.520574 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.520864 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.520868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.520871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.521147 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:52.521303 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:52.521674 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 11:38:52.521856 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":389,"thread_start_us":150,"threads_started":1}
I20260926 11:38:52.523326 16473 tablet_service.cc:1514] Processing DeleteTablet for tablet aa71ae87047e445ab45b89eb62a16a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37106
I20260926 11:38:52.524318 16548 tablet_replica.cc:330] stopping tablet replica
I20260926 11:38:52.524662 16548 raft_consensus.cc:2229] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:38:52.524941 16548 raft_consensus.cc:2258] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:52.526031 16548 ts_tablet_manager.cc:1905] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:52.530388 16548 ts_tablet_manager.cc:1918] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:52.530486 16548 log.cc:1196] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal/wals/aa71ae87047e445ab45b89eb62a16a14
I20260926 11:38:52.530776 16548 ts_tablet_manager.cc:1939] T aa71ae87047e445ab45b89eb62a16a14 P 62534bea9f904d07a2d86c334d169a34: Deleting consensus metadata
I20260926 11:38:52.532950 16337 catalog_manager.cc:4841] TS 62534bea9f904d07a2d86c334d169a34 (127.0.98.1:44635): tablet aa71ae87047e445ab45b89eb62a16a14 (table test-table [id=49ca012c34874207922f8eb1c43f8813]) successfully deleted
W20260926 11:38:52.629138 16543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:52.629518 16543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:52.629575 16543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:52.634207 16543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:52.634311 16543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:52.634382 16543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:38:52.640676 16543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39181
--tserver_master_addrs=127.0.98.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:52.642125 16543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:52.643394 16543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:52.656394 16556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:52.656402 16554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:52.656394 16553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:52.656844 16543 server_base.cc:1034] running on GCE node
I20260926 11:38:52.657459 16543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:52.658970 16543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:52.660317 16543 hybrid_clock.cc:648] HybridClock initialized: now 1790422732660266 us; error 78 us; skew 500 ppm
Sep 26 11:38:52 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:52.664171 16543 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:38:52.665469 16543 webserver.cc:460] Webserver started at http://127.0.98.2:39181/ using document root <none> and password file <none>
I20260926 11:38:52.666174 16543 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:52.666268 16543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:52.671960 16543 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 11:38:52.675285 16563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:52.677256 16543 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:38:52.677488 16543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "5abe887d28ef40cc8c41d1673b7a5b7b"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "7b9c244b291d51669be3de4977840fea"
server_key_iv: "37e98dd1cfbea95d81cf1187a18f0c1a"
server_key_version: "encryptionkey@0"
I20260926 11:38:52.678315 16543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:52.701087 16543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:52.705163 16543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:52.705431 16543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:52.706241 16543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:52.707412 16543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:52.707466 16543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:52.707547 16543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:52.707626 16543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 11:38:52.719785 16543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38317
I20260926 11:38:52.719827 16676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38317 every 8 connection(s)
I20260926 11:38:52.721138 16543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:38:52.730068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 16543
I20260926 11:38:52.730618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41131
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40579
--webserver_port=44385
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:38:52 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:52.739464 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.723627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53683 (local address 127.0.98.62:40579)
0926 11:38:52.723920 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:52.723925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:52.725043 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 11:38:52.726728 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:52.726732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:52.726813 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:52.726941 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:52.728780 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.729380 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.730761 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.731023 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.731146 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.734778 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:52.734782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:52.734784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:52.734817 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:52.736879 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.737562 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.737566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.737568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.737679 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.738099 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.738103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.738105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.738339 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:52.738455 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:52.739092 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 11:38:52.739265 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260926 11:38:52.740608 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.724174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:53683)
0926 11:38:52.724834 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:52.724891 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:52.726417 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:52.727228 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:52.727239 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:52.727833 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:52.728589 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:52.728608 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.729525 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.729530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:52.730621 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:52.730632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.731229 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.731234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:52.731427 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.732286 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:52.732317 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:52.734527 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:52.737050 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:52.737065 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:52.737086 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:52.737420 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:52.737798 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:52.737801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:52.737802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:52.737980 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:52.738454 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:52.738464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:52.738805 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 11:38:52.739031 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 11:38:52.739478 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":175,"threads_started":1}
I20260926 11:38:52.742256 16677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40579
I20260926 11:38:52.742581 16677 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:52.743230 16677 heartbeater.cc:511] Master 127.0.98.62:40579 requested a full tablet report, sending...
I20260926 11:38:52.744880 16345 ts_manager.cc:194] Registered new tserver with Master: 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317)
I20260926 11:38:52.745819 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53683
W20260926 11:38:52.862185 16680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:38:52.862491 16680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:38:52.862536 16680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:38:52.867204 16680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:38:52.867286 16680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:38:52.867350 16680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:38:52.873657 16680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44385
--tserver_master_addrs=127.0.98.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:38:52.874891 16680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:38:52.876040 16680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:38:52.887270 16685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:52.887270 16688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:38:52.887297 16686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:38:52.887692 16680 server_base.cc:1034] running on GCE node
I20260926 11:38:52.888267 16680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:38:52.889230 16680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:38:52.890450 16680 hybrid_clock.cc:648] HybridClock initialized: now 1790422732890433 us; error 45 us; skew 500 ppm
Sep 26 11:38:52 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:38:52.893772 16680 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:38:52.894721 16680 webserver.cc:460] Webserver started at http://127.0.98.3:44385/ using document root <none> and password file <none>
I20260926 11:38:52.895330 16680 fs_manager.cc:359] Metadata directory not provided
I20260926 11:38:52.895406 16680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:38:52.900342 16680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:52.903002 16695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:52.904490 16680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:38:52.904608 16680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "773102a5cfae4965949f7eb6695fc8e5"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "2243f7e08a30b79264e90591ed4cf0bc"
server_key_iv: "d84a2c64899012a345e7a6df83c40bcc"
server_key_version: "encryptionkey@0"
I20260926 11:38:52.905210 16680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:38:52.927388 16680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:38:52.930791 16680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:38:52.930960 16680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:38:52.931510 16680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:38:52.932602 16680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:38:52.932651 16680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:52.932694 16680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:38:52.932709 16680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:38:52.942528 16680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41131
I20260926 11:38:52.942556 16808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41131 every 8 connection(s)
I20260926 11:38:52.943702 16680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:38:52.951272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 16680
Sep 26 11:38:52 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:38:52.959563 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.945769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53787 (local address 127.0.98.62:40579)
0926 11:38:52.945968 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:52.945974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:52.947261 (+ 1287us) server_negotiation.cc:407] Connection header received
0926 11:38:52.948346 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:52.948349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:52.948406 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:52.948501 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:52.949919 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.950453 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.952146 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.952466 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.952594 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.955278 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:52.955286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:52.955289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:52.955311 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:52.956931 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.957567 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.957574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.957576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.957678 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.958131 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.958137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.958139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.958350 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:52.958480 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:52.959204 (+ 724us) server_negotiation.cc:299] Negotiation successful
0926 11:38:52.959346 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 11:38:52.960562 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.952153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.0.98.62:40579)
0926 11:38:52.952398 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:52.952402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:52.952536 (+ 134us) server_negotiation.cc:407] Connection header received
0926 11:38:52.952688 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:52.952691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:52.952735 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:52.952805 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:52.953767 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.954286 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.955269 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.955504 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.955657 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.957110 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:52.957112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:52.957114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:52.957132 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:52.958554 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.959077 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.959080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.959081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.959149 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.959470 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.959475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.959482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.959731 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:52.959950 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:52.960286 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 11:38:52.960401 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":69,"threads_started":1}
I20260926 11:38:52.960677 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.946163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:53787)
0926 11:38:52.947074 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:52.947125 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:52.948114 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:52.948813 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:52.948821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:52.949225 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:38:52.949747 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:52.949760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.950578 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.950580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:52.951963 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:52.951973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.952665 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.952672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:52.952898 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.953571 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:38:52.953597 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:38:52.955086 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:38:52.957075 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:52.957085 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:52.957106 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:52.957436 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:52.957801 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:38:52.957805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:38:52.957807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:38:52.958009 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:38:52.958487 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:38:52.958498 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:38:52.958902 (+ 404us) client_negotiation.cc:769] Sending connection context
0926 11:38:52.959156 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 11:38:52.959543 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":541,"threads_started":1}
I20260926 11:38:52.962103 16809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40579
I20260926 11:38:52.962363 16809 heartbeater.cc:464] Registering TS with master...
I20260926 11:38:52.962896 16809 heartbeater.cc:511] Master 127.0.98.62:40579 requested a full tablet report, sending...
I20260926 11:38:52.963948 16345 ts_manager.cc:194] Registered new tserver with Master: 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131)
I20260926 11:38:52.964462 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53787
I20260926 11:38:52.974560 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:38:52.979058 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.975265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.0.98.62:40579)
0926 11:38:52.975465 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:52.975469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:52.975483 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:38:52.975843 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:52.975845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:52.975960 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:52.976049 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:38:52.976327 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.976850 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.977856 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.978096 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.978187 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.978532 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:38:52.978684 (+ 152us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 11:38:52.978946 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46058: token expired
Metrics: {"server-negotiator.queue_time_us":122}
W20260926 11:38:52.979178 16813 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46058: token expired
W20260926 11:38:52.979537 16280 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:40579)
I20260926 11:38:52.988008 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:52.980012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.0.98.62:40579)
0926 11:38:52.980127 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:52.980131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:52.980151 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:38:52.980268 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:52.980270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:52.980307 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:52.980377 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:38:52.980978 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.981464 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.982383 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:52.982610 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:52.982692 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:52.984541 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:38:52.984544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:38:52.984546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:38:52.984569 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:38:52.986204 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.986728 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.986731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.986733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.986824 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:38:52.987099 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:38:52.987101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:38:52.987102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:38:52.987300 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:38:52.987408 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:52.987763 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 11:38:52.987877 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:38:52.990790 16345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:38:52.994767 16345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:38:53.004782 16473 tablet_service.cc:1467] Processing CreateTablet for tablet 6787079438aa4ea69277804f285aaa4b (DEFAULT_TABLE table=test-table [id=c94f20ac66ff46a0b1addba82f9e7d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:53.005474 16473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6787079438aa4ea69277804f285aaa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:53.008750 16821 tablet_bootstrap.cc:492] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Bootstrap starting.
I20260926 11:38:53.009509 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.003097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41131 (local address 127.0.0.1:46506)
0926 11:38:53.003251 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:53.003264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:53.003336 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:53.004344 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:53.004348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:53.004382 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:53.004546 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:53.004556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.006208 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.006214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:53.007727 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:53.007739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.009011 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.009015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:53.009110 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:53.009173 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 11:38:53.009265 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 11:38:53.009368 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 11:38:53.009588 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.003299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.0.98.3:41131)
0926 11:38:53.003754 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:53.003774 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:53.003799 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:38:53.003884 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:53.003889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:53.004157 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:53.004491 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:53.004831 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.006104 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.007882 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.008900 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.009022 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:53.009200 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:53.009295 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 11:38:53.009415 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
I20260926 11:38:53.010360 16821 tablet_bootstrap.cc:654] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:53.010850 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.003900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38317 (local address 127.0.0.1:35884)
0926 11:38:53.004884 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:53.004901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:53.004992 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:53.005855 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:53.005858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:53.005877 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:53.006003 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:53.006010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.007632 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.007635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:53.009167 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:53.009175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.010406 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.010408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:53.010534 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:53.010560 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:38:53.010640 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 11:38:53.010724 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":117,"threads_started":1}
I20260926 11:38:53.010931 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.004324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.0.98.2:38317)
0926 11:38:53.005165 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:53.005179 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:53.005205 (+ 26us) server_negotiation.cc:407] Connection header received
0926 11:38:53.005295 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:53.005301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:53.005686 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:53.005888 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:53.006112 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.007464 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.009412 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.010163 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.010230 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:53.010327 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:53.010692 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 11:38:53.010784 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":157,"threads_started":1}
I20260926 11:38:53.013178 16821 tablet_bootstrap.cc:492] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: No bootstrap required, opened a new log
I20260926 11:38:53.013285 16821 ts_tablet_manager.cc:1397] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 11:38:53.012964 16743 tablet_service.cc:1467] Processing CreateTablet for tablet 6787079438aa4ea69277804f285aaa4b (DEFAULT_TABLE table=test-table [id=c94f20ac66ff46a0b1addba82f9e7d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:53.013767 16821 raft_consensus.cc:348] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:53.013914 16821 raft_consensus.cc:374] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:53.013965 16821 raft_consensus.cc:729] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62534bea9f904d07a2d86c334d169a34, State: Initialized, Role: FOLLOWER
I20260926 11:38:53.014110 16821 consensus_queue.cc:260] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:53.014283 16743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6787079438aa4ea69277804f285aaa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:53.014062 16611 tablet_service.cc:1467] Processing CreateTablet for tablet 6787079438aa4ea69277804f285aaa4b (DEFAULT_TABLE table=test-table [id=c94f20ac66ff46a0b1addba82f9e7d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:38:53.014966 16821 ts_tablet_manager.cc:1428] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:38:53.015043 16611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6787079438aa4ea69277804f285aaa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:38:53.022308 16824 tablet_bootstrap.cc:492] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: Bootstrap starting.
I20260926 11:38:53.023977 16825 tablet_bootstrap.cc:492] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: Bootstrap starting.
I20260926 11:38:53.025336 16824 tablet_bootstrap.cc:654] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:53.026335 16824 log.cc:824] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:53.026516 16825 tablet_bootstrap.cc:654] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: Neither blocks nor log segments found. Creating new log.
I20260926 11:38:53.027912 16825 log.cc:824] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:38:53.028923 16824 tablet_bootstrap.cc:492] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: No bootstrap required, opened a new log
I20260926 11:38:53.029137 16824 ts_tablet_manager.cc:1397] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 11:38:53.030081 16825 tablet_bootstrap.cc:492] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: No bootstrap required, opened a new log
I20260926 11:38:53.030292 16825 ts_tablet_manager.cc:1397] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:38:53.033730 16824 raft_consensus.cc:348] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:53.034081 16824 raft_consensus.cc:374] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:53.034157 16824 raft_consensus.cc:729] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abe887d28ef40cc8c41d1673b7a5b7b, State: Initialized, Role: FOLLOWER
I20260926 11:38:53.034461 16825 raft_consensus.cc:348] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:53.034731 16825 raft_consensus.cc:374] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:38:53.034693 16824 consensus_queue.cc:260] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:53.034788 16825 raft_consensus.cc:729] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773102a5cfae4965949f7eb6695fc8e5, State: Initialized, Role: FOLLOWER
I20260926 11:38:53.035298 16825 consensus_queue.cc:260] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:53.035867 16824 ts_tablet_manager.cc:1428] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 11:38:53.036075 16809 heartbeater.cc:503] Master 127.0.98.62:40579 was elected leader, sending a full tablet report...
I20260926 11:38:53.036087 16677 heartbeater.cc:503] Master 127.0.98.62:40579 was elected leader, sending a full tablet report...
I20260926 11:38:53.036293 16825 ts_tablet_manager.cc:1428] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260926 11:38:53.194803 16810 tablet.cc:2369] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:38:53.222587 16678 tablet.cc:2369] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:38:53.223982 16540 tablet.cc:2369] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:38:53.242492 16828 raft_consensus.cc:482] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:38:53.242760 16828 raft_consensus.cc:504] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:53.244048 16828 leader_election.cc:288] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131), 62534bea9f904d07a2d86c334d169a34 (127.0.98.1:44635)
I20260926 11:38:53.248652 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.244289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41131 (local address 127.0.98.2:52075)
0926 11:38:53.244771 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:53.244783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:53.244871 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:53.245304 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:53.245307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:53.245328 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:53.245478 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:53.245483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.246449 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.246452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:53.247405 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:53.247415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.248188 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.248192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:53.248301 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:53.248329 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 11:38:53.248398 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 11:38:53.248478 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":258,"thread_start_us":56,"threads_started":1}
I20260926 11:38:53.248695 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.244479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52075 (local address 127.0.98.3:41131)
0926 11:38:53.244611 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:53.244617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:53.244870 (+ 253us) server_negotiation.cc:407] Connection header received
0926 11:38:53.245047 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:53.245051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:53.245097 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:53.245186 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:53.245599 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.246326 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.247532 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.248054 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.248157 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:53.248422 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:53.248485 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 11:38:53.248562 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:38:53.249390 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.244286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44635 (local address 127.0.98.2:36261)
0926 11:38:53.244686 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:53.244702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:38:53.244813 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:38:53.245182 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:38:53.245185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:38:53.245203 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:38:53.245332 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:53.245338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.246433 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.246437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:53.247858 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:38:53.247870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.248977 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.248981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:38:53.249090 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:53.249113 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:38:53.249181 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 11:38:53.249249 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":70,"threads_started":1}
I20260926 11:38:53.249485 16763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6787079438aa4ea69277804f285aaa4b" candidate_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773102a5cfae4965949f7eb6695fc8e5" is_pre_election: true
I20260926 11:38:53.249790 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.244419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36261 (local address 127.0.98.1:44635)
0926 11:38:53.244759 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:53.244764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:53.244790 (+ 26us) server_negotiation.cc:407] Connection header received
0926 11:38:53.244953 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:53.244957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:53.245024 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:53.245150 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:38:53.245463 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.246303 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.248003 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.248660 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.248778 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:53.248945 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:53.249424 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 11:38:53.249666 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":85,"threads_started":1}
I20260926 11:38:53.249832 16763 raft_consensus.cc:2454] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5abe887d28ef40cc8c41d1673b7a5b7b in term 0.
I20260926 11:38:53.250362 16565 leader_election.cc:302] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5abe887d28ef40cc8c41d1673b7a5b7b, 773102a5cfae4965949f7eb6695fc8e5; no voters:
I20260926 11:38:53.250480 16493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6787079438aa4ea69277804f285aaa4b" candidate_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62534bea9f904d07a2d86c334d169a34" is_pre_election: true
I20260926 11:38:53.250710 16828 raft_consensus.cc:2781] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:38:53.250747 16493 raft_consensus.cc:2454] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5abe887d28ef40cc8c41d1673b7a5b7b in term 0.
I20260926 11:38:53.250813 16828 raft_consensus.cc:482] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:38:53.250870 16828 raft_consensus.cc:3037] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:53.252017 16828 raft_consensus.cc:504] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:53.252441 16828 leader_election.cc:288] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [CANDIDATE]: Term 1 election: Requested vote from peers 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131), 62534bea9f904d07a2d86c334d169a34 (127.0.98.1:44635)
I20260926 11:38:53.252836 16763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6787079438aa4ea69277804f285aaa4b" candidate_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773102a5cfae4965949f7eb6695fc8e5"
I20260926 11:38:53.252857 16493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6787079438aa4ea69277804f285aaa4b" candidate_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62534bea9f904d07a2d86c334d169a34"
I20260926 11:38:53.252971 16763 raft_consensus.cc:3037] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:53.252988 16493 raft_consensus.cc:3037] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:38:53.253984 16763 raft_consensus.cc:2454] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5abe887d28ef40cc8c41d1673b7a5b7b in term 1.
I20260926 11:38:53.254280 16493 raft_consensus.cc:2454] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5abe887d28ef40cc8c41d1673b7a5b7b in term 1.
I20260926 11:38:53.254333 16565 leader_election.cc:302] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5abe887d28ef40cc8c41d1673b7a5b7b, 773102a5cfae4965949f7eb6695fc8e5; no voters:
I20260926 11:38:53.254519 16828 raft_consensus.cc:2781] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:38:53.254894 16828 raft_consensus.cc:686] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 1 LEADER]: Becoming Leader. State: Replica: 5abe887d28ef40cc8c41d1673b7a5b7b, State: Running, Role: LEADER
I20260926 11:38:53.255291 16828 consensus_queue.cc:237] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:38:53.258755 16351 catalog_manager.cc:5495] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } health_report { overall_health: UNKNOWN } } }
I20260926 11:38:53.310019 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.305946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.0.98.2:38317)
0926 11:38:53.306088 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:38:53.306093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:38:53.306104 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:38:53.306182 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:38:53.306184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:38:53.306238 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:38:53.306344 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:38:53.306638 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.307231 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.308239 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:38:53.308513 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:38:53.308621 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:38:53.308979 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:38:53.309505 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:38:53.309608 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:38:53.309757 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 11:38:53.309817 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:38:53.317119 16493 raft_consensus.cc:1261] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Refusing update from remote peer 5abe887d28ef40cc8c41d1673b7a5b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:53.317121 16763 raft_consensus.cc:1261] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [term 1 FOLLOWER]: Refusing update from remote peer 5abe887d28ef40cc8c41d1673b7a5b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:38:53.318014 16833 consensus_queue.cc:1035] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:53.318570 16833 consensus_queue.cc:1035] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:38:53.328919 16836 mvcc.cc:205] Tried to move back new op lower bound from 7333571515657306112 to 7333571515416743936. Current Snapshot: MvccSnapshot[applied={T|T < 7333571515657306112 or (T in {7333571515657306112})}]
I20260926 11:38:53.333292 16345 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:53.333504 16345 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:38:53.336432 16345 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Sending DeleteTablet for 3 replicas of tablet 6787079438aa4ea69277804f285aaa4b
I20260926 11:38:53.337257 16743 tablet_service.cc:1514] Processing DeleteTablet for tablet 6787079438aa4ea69277804f285aaa4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46506
I20260926 11:38:53.337314 16611 tablet_service.cc:1514] Processing DeleteTablet for tablet 6787079438aa4ea69277804f285aaa4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35884
I20260926 11:38:53.337829 16846 tablet_replica.cc:330] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: stopping tablet replica
I20260926 11:38:53.338038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 16389
I20260926 11:38:53.338083 16847 tablet_replica.cc:330] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: stopping tablet replica
I20260926 11:38:53.338343 16846 raft_consensus.cc:2229] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:38:53.338851 16847 raft_consensus.cc:2229] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:38:53.339226 16847 raft_consensus.cc:2258] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:53.339476 16846 raft_consensus.cc:2258] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:38:53.340970 16847 ts_tablet_manager.cc:1905] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:53.340970 16846 ts_tablet_manager.cc:1905] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:38:53.343843 16846 ts_tablet_manager.cc:1918] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:53.343845 16847 ts_tablet_manager.cc:1918] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:38:53.343958 16847 log.cc:1196] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/6787079438aa4ea69277804f285aaa4b
I20260926 11:38:53.344102 16846 log.cc:1196] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/6787079438aa4ea69277804f285aaa4b
I20260926 11:38:53.344313 16847 ts_tablet_manager.cc:1939] T 6787079438aa4ea69277804f285aaa4b P 5abe887d28ef40cc8c41d1673b7a5b7b: Deleting consensus metadata
I20260926 11:38:53.344626 16846 ts_tablet_manager.cc:1939] T 6787079438aa4ea69277804f285aaa4b P 773102a5cfae4965949f7eb6695fc8e5: Deleting consensus metadata
I20260926 11:38:53.345748 16336 catalog_manager.cc:4841] TS 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131): tablet 6787079438aa4ea69277804f285aaa4b (table test-table [id=c94f20ac66ff46a0b1addba82f9e7d9c]) successfully deleted
I20260926 11:38:53.345788 16335 catalog_manager.cc:4841] TS 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317): tablet 6787079438aa4ea69277804f285aaa4b (table test-table [id=c94f20ac66ff46a0b1addba82f9e7d9c]) successfully deleted
W20260926 11:38:53.348789 16337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44635 (error 108)
I20260926 11:38:53.349267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 16543
I20260926 11:38:53.350301 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:38:53.349903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44635 (local address 127.0.0.1:37116)
0926 11:38:53.350094 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 11:38:53.350185 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 11:38:53.350492 16337 catalog_manager.cc:4584] TS 62534bea9f904d07a2d86c334d169a34 (127.0.98.1:44635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6787079438aa4ea69277804f285aaa4b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44635: connect: Connection refused (error 111)
I20260926 11:38:53.355973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 16680
I20260926 11:38:53.361850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 16318
I20260926 11:39:13.369709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40579
--webserver_interface=127.0.98.62
--webserver_port=32895
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:13.503748 16849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:13.504107 16849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:13.504154 16849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:13.508577 16849 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 11:39:13.508652 16849 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 11:39:13.508669 16849 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 11:39:13.508684 16849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:13.508704 16849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:13.508724 16849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:13.508740 16849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:13.508756 16849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:13.514815 16849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:13.516253 16849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:13.517557 16849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:13.528493 16857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:13.528486 16854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:13.528510 16855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:13.529863 16849 server_base.cc:1034] running on GCE node
I20260926 11:39:13.530612 16849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:13.532241 16849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:13.533440 16849 hybrid_clock.cc:648] HybridClock initialized: now 1790422753533424 us; error 44 us; skew 500 ppm
Sep 26 11:39:13 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:13.537484 16849 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:39:13.538852 16849 webserver.cc:460] Webserver started at http://127.0.98.62:32895/ using document root <none> and password file <none>
I20260926 11:39:13.539669 16849 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:13.539760 16849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:13.545428 16849 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:13.548808 16864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:13.550765 16849 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:13.550941 16849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "070eca456a2f4757a11f2e36a093dda9"
format_stamp: "Formatted at 2026-09-26 11:38:30 on dist-test-slave-4pw5"
server_key: "82502e8c2c2482320f2dd37e7f6cc678"
server_key_iv: "751921f512c14f34b146f3f413047741"
server_key_version: "encryptionkey@0"
I20260926 11:39:13.551827 16849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:13.570621 16849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:13.574671 16849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:13.574916 16849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:13.584760 16849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40579
I20260926 11:39:13.585131 16916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40579 every 8 connection(s)
I20260926 11:39:13.586241 16849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:13.589003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 16849
I20260926 11:39:13.589540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44635
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40579
--webserver_port=42239
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:39:13.591989 16917 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:39:13.594022 16917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Bootstrap starting.
I20260926 11:39:13.612754 16917 log.cc:824] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:13.625123 16917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:39:13.625629 16917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Bootstrap complete.
I20260926 11:39:13.630707 16917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:39:13.631250 16917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070eca456a2f4757a11f2e36a093dda9, State: Initialized, Role: FOLLOWER
I20260926 11:39:13.631953 16917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:39:13.632146 16917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:13.632247 16917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:13.632396 16917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 2 FOLLOWER]: Advancing to term 3
I20260926 11:39:13.634773 16917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:39:13.635210 16917 leader_election.cc:302] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070eca456a2f4757a11f2e36a093dda9; no voters:
I20260926 11:39:13.635601 16917 leader_election.cc:288] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 11:39:13.635869 16921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 11:39:13.636221 16921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [term 3 LEADER]: Becoming Leader. State: Replica: 070eca456a2f4757a11f2e36a093dda9, State: Running, Role: LEADER
I20260926 11:39:13.636632 16921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } }
I20260926 11:39:13.637190 16917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:13.638587 16923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070eca456a2f4757a11f2e36a093dda9. Latest consensus state: current_term: 3 leader_uuid: "070eca456a2f4757a11f2e36a093dda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } } }
I20260926 11:39:13.638688 16922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "070eca456a2f4757a11f2e36a093dda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070eca456a2f4757a11f2e36a093dda9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40579 } } }
I20260926 11:39:13.638830 16922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:13.638765 16923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:13.639441 16931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:13.646626 16931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:13.647465 16931 catalog_manager.cc:1259] Loaded cluster ID: e5731ba5edfe4066ab70091a01bcdfd4
I20260926 11:39:13.647539 16931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:13.650427 16931 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:13.651325 16931 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Loaded TSK: 0
I20260926 11:39:13.651424 16931 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 070eca456a2f4757a11f2e36a093dda9: Loaded TSK: 1
F20260926 11:39:13.652328 16931 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f03030809ad google::LogMessage::Fail() at ??:0
@ 0x7f030308388a google::LogMessage::SendToLog() at ??:0
@ 0x7f03030804de google::LogMessage::Flush() at ??:0
@ 0x7f0303081a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0305d4418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0305d3cf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f030d9e57bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f030d9bf99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f030d9c0178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f030da01df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f030da2082b std::function<>::operator()() at ??:0
@ 0x7f030d9bfb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f030d9c0f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f030d9bd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f030da01640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f03052d23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f03052d2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f03052d43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f03052c3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f03069076db start_thread at ??:0
@ 0x7f0301b7e61f clone at ??:0
W20260926 11:39:13.768293 16919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:13.768837 16919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:13.768923 16919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:13.776103 16919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:13.776255 16919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:13.776407 16919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:39:13.786087 16919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42239
--tserver_master_addrs=127.0.98.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:13.788331 16919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:13.790231 16919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:13.804543 16967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:13.804563 16964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:13.805051 16965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:13.805183 16919 server_base.cc:1034] running on GCE node
I20260926 11:39:13.805909 16919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:13.806885 16919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:13.808193 16919 hybrid_clock.cc:648] HybridClock initialized: now 1790422753808152 us; error 70 us; skew 500 ppm
Sep 26 11:39:13 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:13.811892 16919 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:39:13.813084 16919 webserver.cc:460] Webserver started at http://127.0.98.1:42239/ using document root <none> and password file <none>
I20260926 11:39:13.813798 16919 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:13.813911 16919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:13.819303 16919 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 11:39:13.822479 16974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:13.824349 16919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 11:39:13.824536 16919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "62534bea9f904d07a2d86c334d169a34"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "9e8d46ba7f9c71dc74d7ba89267581b7"
server_key_iv: "460373a4922ece0b62a06e70846f069e"
server_key_version: "encryptionkey@0"
I20260926 11:39:13.825203 16919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:13.849931 16919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:13.854404 16919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:13.854676 16919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:13.855419 16919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:13.857084 16981 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:39:13.864650 16919 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:39:13.864816 16919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260926 11:39:13.864928 16919 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:39:13.867213 16919 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:39:13.867309 16919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 11:39:13.867861 16981 tablet_bootstrap.cc:492] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Bootstrap starting.
I20260926 11:39:13.879438 16919 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44635
I20260926 11:39:13.879629 17088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44635 every 8 connection(s)
I20260926 11:39:13.880919 16919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:39:13.889689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 16919
I20260926 11:39:13.890295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38317
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40579
--webserver_port=39181
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:39:13.893451 16981 log.cc:824] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:13.906592 16981 tablet_bootstrap.cc:492] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:39:13.907171 16981 tablet_bootstrap.cc:492] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Bootstrap complete.
I20260926 11:39:13.908797 16981 ts_tablet_manager.cc:1397] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Time spent bootstrapping tablet: real 0.041s user 0.032s sys 0.007s
I20260926 11:39:13.913005 16981 raft_consensus.cc:348] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:39:13.913695 16981 raft_consensus.cc:729] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62534bea9f904d07a2d86c334d169a34, State: Initialized, Role: FOLLOWER
I20260926 11:39:13.914348 16981 consensus_queue.cc:260] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:39:13.915753 16981 ts_tablet_manager.cc:1428] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
W20260926 11:39:13.935527 17091 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:39:13.884385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:54217)
0926 11:39:13.885254 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:13.885375 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:13.886942 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:13.934416 (+ 47474us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":205,"threads_started":1}
W20260926 11:39:13.936354 17089 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40579 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40579: Client connection negotiation failed: client connection to 127.0.98.62:40579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 11:39:13.937395 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:13.937044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:37503)
0926 11:39:13.937220 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:13.937311 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 11:39:13.938257 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:13.938014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:54429)
0926 11:39:13.938140 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:13.938194 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 11:39:13.938583 17089 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:39:14.036100 17092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:14.036527 17092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:14.036607 17092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:14.041097 17092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:14.041222 17092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:14.041329 17092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:39:14.047402 17092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39181
--tserver_master_addrs=127.0.98.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:14.048961 17092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:14.050349 17092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:14.062803 17100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:14.062783 17099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:14.062783 17102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:14.063295 17092 server_base.cc:1034] running on GCE node
I20260926 11:39:14.063905 17092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:14.064831 17092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:14.066058 17092 hybrid_clock.cc:648] HybridClock initialized: now 1790422754066029 us; error 55 us; skew 500 ppm
Sep 26 11:39:14 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:14.069571 17092 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:39:14.070878 17092 webserver.cc:460] Webserver started at http://127.0.98.2:39181/ using document root <none> and password file <none>
I20260926 11:39:14.071666 17092 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:14.071746 17092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:14.077093 17092 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:14.080123 17109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:14.082053 17092 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 11:39:14.082248 17092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "5abe887d28ef40cc8c41d1673b7a5b7b"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "7b9c244b291d51669be3de4977840fea"
server_key_iv: "37e98dd1cfbea95d81cf1187a18f0c1a"
server_key_version: "encryptionkey@0"
I20260926 11:39:14.083001 17092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:14.107975 17092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:14.112310 17092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:14.112567 17092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:14.113385 17092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:14.114470 17092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:14.114522 17092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:14.114598 17092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:14.114634 17092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:14.127079 17092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38317
I20260926 11:39:14.127104 17222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38317 every 8 connection(s)
I20260926 11:39:14.128500 17092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:39:14.131287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 17092
I20260926 11:39:14.131910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41131
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40579
--webserver_port=44385
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:39:14.134613 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:14.131885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:48645)
0926 11:39:14.132664 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:14.133430 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":184,"threads_started":1}
W20260926 11:39:14.135257 17223 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40579 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40579: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
I20260926 11:39:14.136236 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:14.135855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:57831)
0926 11:39:14.136054 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:14.136142 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 11:39:14.137243 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:14.136940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:53397)
0926 11:39:14.137093 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:14.137161 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 11:39:14.137656 17223 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:39:14.264881 17226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:14.265208 17226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:14.265255 17226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:14.270298 17226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:14.270393 17226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:14.270471 17226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:39:14.276850 17226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44385
--tserver_master_addrs=127.0.98.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:14.278117 17226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:14.279294 17226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:14.290800 17232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:14.290820 17231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:14.290820 17234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:14.291316 17226 server_base.cc:1034] running on GCE node
I20260926 11:39:14.292081 17226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:14.293016 17226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:14.294268 17226 hybrid_clock.cc:648] HybridClock initialized: now 1790422754294249 us; error 40 us; skew 500 ppm
Sep 26 11:39:14 dist-test-slave-4pw5 krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:14.297863 17226 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:39:14.299111 17226 webserver.cc:460] Webserver started at http://127.0.98.3:44385/ using document root <none> and password file <none>
I20260926 11:39:14.299927 17226 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:14.300005 17226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:14.305344 17226 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:14.308528 17241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:14.310536 17226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:14.310734 17226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "773102a5cfae4965949f7eb6695fc8e5"
format_stamp: "Formatted at 2026-09-26 11:38:31 on dist-test-slave-4pw5"
server_key: "2243f7e08a30b79264e90591ed4cf0bc"
server_key_iv: "d84a2c64899012a345e7a6df83c40bcc"
server_key_version: "encryptionkey@0"
I20260926 11:39:14.311491 17226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:14.325759 17226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:14.330441 17226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:14.330703 17226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:14.331483 17226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:14.332568 17226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:14.332623 17226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:14.332669 17226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:14.332684 17226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:14.343807 17226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41131
I20260926 11:39:14.343828 17354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41131 every 8 connection(s)
I20260926 11:39:14.345002 17226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:39:14.349192 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:14.347502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:43347)
0926 11:39:14.348316 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:14.348448 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":121,"threads_started":1}
W20260926 11:39:14.349783 17355 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40579 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40579: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
I20260926 11:39:14.350567 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:14.350284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:60909)
0926 11:39:14.350439 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:14.350509 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 11:39:14.351253 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:14.351034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:33819)
0926 11:39:14.351146 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:14.351195 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 11:39:14.351454 17355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 11:39:14.351711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 17226
I20260926 11:39:14.940049 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:14.939384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:43857)
0926 11:39:14.939775 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:14.939842 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":78,"threads_started":1}
I20260926 11:39:15.138849 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:15.138370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:37597)
0926 11:39:15.138647 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:15.138711 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260926 11:39:15.352635 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:15.352132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:38105)
0926 11:39:15.352452 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:15.352516 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20260926 11:39:15.661041 17362 raft_consensus.cc:482] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:15.661271 17362 raft_consensus.cc:504] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:39:15.662412 17362 leader_election.cc:288] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131), 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317)
Sep 26 11:39:15 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 26 11:39:15 dist-test-slave-4pw5 krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 11:39:15.677378 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:15.662723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38317 (local address 127.0.98.1:53537)
0926 11:39:15.663266 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:15.663280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:15.663383 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:15.664935 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:15.664941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:15.665533 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:15.666021 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:15.666037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:15.667368 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:15.667371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:15.668320 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:15.668330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:15.669073 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:15.669077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:15.669332 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:15.670105 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:15.670121 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:15.672101 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:15.674655 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:15.674666 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:15.674686 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:15.675076 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:15.675458 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:15.675461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:15.675462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:15.675643 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:15.676307 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:15.676313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:15.676560 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 11:39:15.676749 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 11:39:15.677159 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":262,"spinlock_wait_cycles":37504,"thread_start_us":121,"threads_started":1}
I20260926 11:39:15.677368 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:15.662876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47567 (local address 127.0.98.3:41131)
0926 11:39:15.663247 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:15.663277 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:15.663326 (+ 49us) server_negotiation.cc:407] Connection header received
0926 11:39:15.664162 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:15.664169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:15.664548 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:15.664878 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:15.666219 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:15.667208 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:15.668472 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:15.668949 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:15.669123 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:15.672513 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:15.672519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:15.672529 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:15.672560 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:15.674595 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:15.675190 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:15.675197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:15.675202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:15.675330 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:15.675733 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:15.675736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:15.675738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:15.676195 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:15.676341 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:15.676872 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 11:39:15.677153 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":80,"threads_started":1}
I20260926 11:39:15.677368 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:15.663034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53537 (local address 127.0.98.2:38317)
0926 11:39:15.663482 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:15.663492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:15.663521 (+ 29us) server_negotiation.cc:407] Connection header received
0926 11:39:15.664207 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:15.664212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:15.664547 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:15.664877 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:15.666228 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:15.667207 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:15.668471 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:15.668941 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:15.669112 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:15.672375 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:15.672386 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:15.672416 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:15.672463 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:15.674503 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:15.675211 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:15.675217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:15.675221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:15.675332 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:15.675763 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:15.675766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:15.675767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:15.676185 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:15.676325 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:15.676872 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 11:39:15.677169 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":128,"threads_started":1}
I20260926 11:39:15.677433 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:15.662723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41131 (local address 127.0.98.1:47567)
0926 11:39:15.663060 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:15.663077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:15.663201 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:15.664935 (+ 1734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:15.664941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:15.665508 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:15.666020 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:15.666037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:15.667364 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:15.667368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:15.668321 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:15.668331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:15.669093 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:15.669097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:15.669312 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:15.670098 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:15.670124 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:15.672308 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:15.674728 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:15.674731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:15.674732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:15.675064 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:15.675432 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:15.675438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:15.675440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:15.675631 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:15.676327 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:15.676331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:15.676559 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 11:39:15.676749 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 11:39:15.677250 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":391,"spinlock_wait_cycles":22656,"thread_start_us":77,"threads_started":1}
W20260926 11:39:15.680239 16975 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
W20260926 11:39:15.680861 16976 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
I20260926 11:39:15.680989 16976 leader_election.cc:302] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62534bea9f904d07a2d86c334d169a34; no voters: 5abe887d28ef40cc8c41d1673b7a5b7b, 773102a5cfae4965949f7eb6695fc8e5
I20260926 11:39:15.681317 17362 raft_consensus.cc:2726] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:39:15.941291 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:15.941021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:54237)
0926 11:39:15.941147 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:15.941203 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 11:39:16.140331 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:16.139779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:37957)
0926 11:39:16.140149 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:16.140212 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260926 11:39:16.353956 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:16.353477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:36521)
0926 11:39:16.353754 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:16.353812 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260926 11:39:16.942603 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:16.942146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:36435)
0926 11:39:16.942430 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:16.942505 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260926 11:39:17.141776 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:17.141295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:55015)
0926 11:39:17.141591 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:17.141661 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260926 11:39:17.355367 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:17.354886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:54885)
0926 11:39:17.355181 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:17.355244 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260926 11:39:17.443727 17372 raft_consensus.cc:482] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:17.443862 17372 raft_consensus.cc:504] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:39:17.444458 17372 leader_election.cc:288] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131), 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317)
W20260926 11:39:17.445436 16975 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
W20260926 11:39:17.445549 16976 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
I20260926 11:39:17.445631 16976 leader_election.cc:302] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62534bea9f904d07a2d86c334d169a34; no voters: 5abe887d28ef40cc8c41d1673b7a5b7b, 773102a5cfae4965949f7eb6695fc8e5
I20260926 11:39:17.445778 17372 raft_consensus.cc:2726] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:39:17.944010 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:17.943514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:54607)
0926 11:39:17.943839 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:17.943905 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":144,"threads_started":1}
I20260926 11:39:18.143350 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:18.142853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:58397)
0926 11:39:18.143181 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:18.143250 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":99,"threads_started":1}
I20260926 11:39:18.356784 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:18.356224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:59627)
0926 11:39:18.356538 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:18.356593 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":85,"threads_started":1}
I20260926 11:39:18.945695 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:18.945131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:60275)
0926 11:39:18.945488 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:18.945549 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260926 11:39:19.144879 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:19.144385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:60091)
0926 11:39:19.144708 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:19.144776 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260926 11:39:19.358598 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:19.357901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:42467)
0926 11:39:19.358331 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:19.358490 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":135,"threads_started":1}
I20260926 11:39:19.947158 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:19.946657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:59655)
0926 11:39:19.946962 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:19.947060 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":81,"threads_started":1}
I20260926 11:39:20.065775 17380 raft_consensus.cc:482] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:20.066008 17380 raft_consensus.cc:504] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:39:20.066520 17380 leader_election.cc:288] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131), 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317)
W20260926 11:39:20.067411 16976 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
W20260926 11:39:20.067574 16975 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
I20260926 11:39:20.067719 16975 leader_election.cc:302] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62534bea9f904d07a2d86c334d169a34; no voters: 5abe887d28ef40cc8c41d1673b7a5b7b, 773102a5cfae4965949f7eb6695fc8e5
I20260926 11:39:20.067945 17380 raft_consensus.cc:2726] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:39:20.146282 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:20.145845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:53703)
0926 11:39:20.146122 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:20.146184 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260926 11:39:20.360321 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:20.359735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:58651)
0926 11:39:20.360158 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:20.360221 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":91,"threads_started":1}
I20260926 11:39:20.948634 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:20.948157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:52021)
0926 11:39:20.948462 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:20.948529 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":80,"threads_started":1}
I20260926 11:39:21.147842 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:21.147224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:55483)
0926 11:39:21.147671 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:21.147738 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":94,"threads_started":1}
I20260926 11:39:21.361909 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:21.361310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:56591)
0926 11:39:21.361671 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:21.361737 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":96,"threads_started":1}
I20260926 11:39:21.950580 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:21.949936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:47383)
0926 11:39:21.950316 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:21.950410 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260926 11:39:22.149595 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:22.149069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:48063)
0926 11:39:22.149385 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:22.149459 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":96,"threads_started":1}
I20260926 11:39:22.363672 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:22.362997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:50763)
0926 11:39:22.363378 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:22.363458 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":108,"threads_started":1}
I20260926 11:39:22.952428 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:22.951740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:36469)
0926 11:39:22.952198 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:22.952288 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":141,"threads_started":1}
I20260926 11:39:23.151343 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:23.150731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:42123)
0926 11:39:23.151107 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:23.151177 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260926 11:39:23.365379 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:23.364819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:33383)
0926 11:39:23.365182 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:23.365246 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":141,"threads_started":1}
I20260926 11:39:23.954185 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:23.953724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:58529)
0926 11:39:23.954015 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:23.954082 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":75,"threads_started":1}
I20260926 11:39:24.152884 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:24.152437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:43745)
0926 11:39:24.152710 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:24.152772 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260926 11:39:24.366858 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:24.366384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:42037)
0926 11:39:24.366704 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:24.366756 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":134,"threads_started":1}
I20260926 11:39:24.806097 17395 raft_consensus.cc:482] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:24.806288 17395 raft_consensus.cc:504] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:39:24.806836 17395 leader_election.cc:288] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131), 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317)
W20260926 11:39:24.807883 16976 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
W20260926 11:39:24.808033 16975 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
I20260926 11:39:24.808163 16975 leader_election.cc:302] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62534bea9f904d07a2d86c334d169a34; no voters: 5abe887d28ef40cc8c41d1673b7a5b7b, 773102a5cfae4965949f7eb6695fc8e5
I20260926 11:39:24.808380 17395 raft_consensus.cc:2726] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:39:24.955884 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:24.955222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:47637)
0926 11:39:24.955688 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:24.955770 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":138,"threads_started":1}
I20260926 11:39:25.154583 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:25.153932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:56357)
0926 11:39:25.154381 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:25.154466 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":128,"threads_started":1}
I20260926 11:39:25.368461 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:25.367958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:56523)
0926 11:39:25.368284 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:25.368355 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":143,"threads_started":1}
I20260926 11:39:25.957819 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:25.957137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:48379)
0926 11:39:25.957570 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:25.957708 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":121,"threads_started":1}
I20260926 11:39:26.156294 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:26.155649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:49889)
0926 11:39:26.156086 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:26.156181 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":98,"threads_started":1}
I20260926 11:39:26.370301 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:26.369644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:40475)
0926 11:39:26.370127 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:26.370199 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":225,"threads_started":1}
I20260926 11:39:26.959501 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:26.958831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:51959)
0926 11:39:26.959247 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:26.959345 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":153,"threads_started":1}
I20260926 11:39:27.157891 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:27.157362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:45599)
0926 11:39:27.157712 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:27.157786 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":101,"threads_started":1}
I20260926 11:39:27.372051 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:27.371283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:48721)
0926 11:39:27.371772 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:27.371877 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":92,"threads_started":1}
I20260926 11:39:27.961381 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:27.960748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:49011)
0926 11:39:27.961180 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:27.961269 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":111,"threads_started":1}
I20260926 11:39:28.159528 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:28.158897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:51327)
0926 11:39:28.159312 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:28.159401 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":111,"threads_started":1}
I20260926 11:39:28.373603 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:28.373077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.3:34125)
0926 11:39:28.373398 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:28.373472 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260926 11:39:28.878578 17408 raft_consensus.cc:482] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:28.878742 17408 raft_consensus.cc:504] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773102a5cfae4965949f7eb6695fc8e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41131 } } peers { permanent_uuid: "5abe887d28ef40cc8c41d1673b7a5b7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38317 } } peers { permanent_uuid: "62534bea9f904d07a2d86c334d169a34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44635 } }
I20260926 11:39:28.879361 17408 leader_election.cc:288] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131), 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317)
W20260926 11:39:28.880367 16976 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 773102a5cfae4965949f7eb6695fc8e5 (127.0.98.3:41131): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
W20260926 11:39:28.880529 16975 leader_election.cc:341] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5abe887d28ef40cc8c41d1673b7a5b7b (127.0.98.2:38317): Not found: Tablet not found: 6787079438aa4ea69277804f285aaa4b
I20260926 11:39:28.880591 16975 leader_election.cc:302] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62534bea9f904d07a2d86c334d169a34; no voters: 5abe887d28ef40cc8c41d1673b7a5b7b, 773102a5cfae4965949f7eb6695fc8e5
I20260926 11:39:28.880901 17408 raft_consensus.cc:2726] T 6787079438aa4ea69277804f285aaa4b P 62534bea9f904d07a2d86c334d169a34 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:39:28.963279 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:28.962726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.1:55665)
0926 11:39:28.963089 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:28.963168 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":122,"threads_started":1}
I20260926 11:39:29.161544 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:29.160830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40579 (local address 127.0.98.2:58715)
0926 11:39:29.161279 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:29.161372 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":166,"threads_started":1}
I20260926 11:39:29.353363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 16919
I20260926 11:39:29.361325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 17092
I20260926 11:39:29.368219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 17226
2026-09-26T11:39:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59853 ms)
[----------] 30 tests from SecurityITest (257696 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:29 dist-test-slave-4pw5 krb5kdc[17413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:29 dist-test-slave-4pw5 krb5kdc[17413](info): set up 2 sockets
Sep 26 11:39:29 dist-test-slave-4pw5 krb5kdc[17413](info): commencing operation
krb5kdc: starting...
W20260926 11:39:30.436085 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:30 dist-test-slave-4pw5 krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:39:30.572190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:30.691808 17426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:30.692198 17426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:30.692284 17426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:30.696597 17426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:30.696698 17426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:30.696735 17426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:30.696792 17426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:30.696830 17426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:30.702417 17426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:30.703697 17426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:30.704752 17426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:30.714926 17431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:30.715039 17432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:30.715240 17426 server_base.cc:1034] running on GCE node
W20260926 11:39:30.715289 17434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:30.715971 17426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:30.717267 17426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:30.718458 17426 hybrid_clock.cc:648] HybridClock initialized: now 1790422770718445 us; error 34 us; skew 500 ppm
Sep 26 11:39:30 dist-test-slave-4pw5 krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:30.721499 17426 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:39:30.722496 17426 webserver.cc:460] Webserver started at http://127.0.98.62:45331/ using document root <none> and password file <none>
I20260926 11:39:30.723088 17426 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:30.723168 17426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:30.723344 17426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:30.725539 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "f7246937463e414db83da32a1a08bef2"
format_stamp: "Formatted at 2026-09-26 11:39:30 on dist-test-slave-4pw5"
server_key: "485ec33f8aa47f1b914b92bbf1703c8b"
server_key_iv: "828efd818391f48fbf6c14f27781ef8e"
server_key_version: "encryptionkey@0"
I20260926 11:39:30.726123 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7246937463e414db83da32a1a08bef2"
format_stamp: "Formatted at 2026-09-26 11:39:30 on dist-test-slave-4pw5"
server_key: "485ec33f8aa47f1b914b92bbf1703c8b"
server_key_iv: "828efd818391f48fbf6c14f27781ef8e"
server_key_version: "encryptionkey@0"
I20260926 11:39:30.730494 17426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:39:30.733948 17441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:30.735538 17426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:39:30.735971 17426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "f7246937463e414db83da32a1a08bef2"
format_stamp: "Formatted at 2026-09-26 11:39:30 on dist-test-slave-4pw5"
server_key: "485ec33f8aa47f1b914b92bbf1703c8b"
server_key_iv: "828efd818391f48fbf6c14f27781ef8e"
server_key_version: "encryptionkey@0"
I20260926 11:39:30.736366 17426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:30.766644 17426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:30.770767 17426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:30.770982 17426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:30.779426 17426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38381
I20260926 11:39:30.779448 17493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38381 every 8 connection(s)
I20260926 11:39:30.780601 17426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:30.784310 17494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:30.789947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 17426
I20260926 11:39:30.790094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:30.790452 392 external_mini_cluster.cc:1428] Setting key 6274e915a08e5531bb61b891db5a16a1
I20260926 11:39:30.792008 17494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2: Bootstrap starting.
I20260926 11:39:30.795516 17494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:30.796463 17494 log.cc:824] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:30.799281 17494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2: No bootstrap required, opened a new log
Sep 26 11:39:30 dist-test-slave-4pw5 krb5kdc[17413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:39:30.803843 17494 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7246937463e414db83da32a1a08bef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } }
I20260926 11:39:30.804301 17494 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:30.804396 17494 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7246937463e414db83da32a1a08bef2, State: Initialized, Role: FOLLOWER
I20260926 11:39:30.805030 17494 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7246937463e414db83da32a1a08bef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } }
I20260926 11:39:30.805217 17494 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:30.805346 17494 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:30.805486 17494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:30.806949 17494 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7246937463e414db83da32a1a08bef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } }
I20260926 11:39:30.807366 17494 leader_election.cc:302] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7246937463e414db83da32a1a08bef2; no voters:
I20260926 11:39:30.807772 17494 leader_election.cc:288] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:30.807868 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:30.792299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.0.98.62:38381)
0926 11:39:30.792786 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:30.792804 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:30.792841 (+ 37us) server_negotiation.cc:407] Connection header received
0926 11:39:30.793566 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:30.793584 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:30.793936 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:30.794260 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:30.795546 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:30.797181 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:30.798600 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:30.799129 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:30.799265 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:30.802288 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:30.802295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:30.802312 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:30.802346 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:30.804379 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:30.804959 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:30.804964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:30.804969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:30.805161 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:30.805549 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:30.805552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:30.805553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:30.806054 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:30.806230 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:30.806520 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 11:39:30.806833 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":167,"threads_started":1}
I20260926 11:39:30.807883 17499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:30.808111 17499 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [term 1 LEADER]: Becoming Leader. State: Replica: f7246937463e414db83da32a1a08bef2, State: Running, Role: LEADER
I20260926 11:39:30.808447 17499 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7246937463e414db83da32a1a08bef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } }
I20260926 11:39:30.809314 17494 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:30.809708 17499 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7246937463e414db83da32a1a08bef2. Latest consensus state: current_term: 1 leader_uuid: "f7246937463e414db83da32a1a08bef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7246937463e414db83da32a1a08bef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } }
I20260926 11:39:30.809860 17499 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:30.809708 17501 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7246937463e414db83da32a1a08bef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7246937463e414db83da32a1a08bef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } }
I20260926 11:39:30.810010 17501 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2 [sys.catalog]: This master's current role is: LEADER
W20260926 11:39:30.814548 17514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:39:30.814649 17514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:39:30.814749 17503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:30.815560 17503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:30.821477 17503 catalog_manager.cc:1347] Generated new cluster ID: c33aa12f33f74f75b8c6e7a118ee9c35
I20260926 11:39:30.821575 17503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:30.843384 17503 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:30.845085 17503 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:30.857443 17503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2: Generated new TSK 0
I20260926 11:39:30.858484 17503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:39:30.899246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38381
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:31.023170 17520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:31.023453 17520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:31.023499 17520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:31.028132 17520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:31.028223 17520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:31.028290 17520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:39:31.033885 17520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:31.035002 17520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:31.036428 17520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:31.046975 17525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:31.046976 17526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:31.047016 17528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:31.047435 17520 server_base.cc:1034] running on GCE node
I20260926 11:39:31.047948 17520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:31.048678 17520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:31.049877 17520 hybrid_clock.cc:648] HybridClock initialized: now 1790422771049859 us; error 42 us; skew 500 ppm
Sep 26 11:39:31 dist-test-slave-4pw5 krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:31.056190 17520 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:39:31.057219 17520 webserver.cc:460] Webserver started at http://127.0.98.1:33913/ using document root <none> and password file <none>
I20260926 11:39:31.057790 17520 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:31.057850 17520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:31.058023 17520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:31.060276 17520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b48b91aacc7495fb51ecefc46f38e83"
format_stamp: "Formatted at 2026-09-26 11:39:31 on dist-test-slave-4pw5"
server_key: "55f27fc7f83ff34db5a19a101ed4c5c5"
server_key_iv: "7ca6d6eba42c514c3b061aa85d322fe7"
server_key_version: "encryptionkey@0"
I20260926 11:39:31.060822 17520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b48b91aacc7495fb51ecefc46f38e83"
format_stamp: "Formatted at 2026-09-26 11:39:31 on dist-test-slave-4pw5"
server_key: "55f27fc7f83ff34db5a19a101ed4c5c5"
server_key_iv: "7ca6d6eba42c514c3b061aa85d322fe7"
server_key_version: "encryptionkey@0"
I20260926 11:39:31.065019 17520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 11:39:31.067853 17535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:31.069514 17520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:39:31.069708 17520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "9b48b91aacc7495fb51ecefc46f38e83"
format_stamp: "Formatted at 2026-09-26 11:39:31 on dist-test-slave-4pw5"
server_key: "55f27fc7f83ff34db5a19a101ed4c5c5"
server_key_iv: "7ca6d6eba42c514c3b061aa85d322fe7"
server_key_version: "encryptionkey@0"
I20260926 11:39:31.069855 17520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:31.086268 17520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:31.090189 17520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:31.090368 17520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:31.090965 17520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:31.091923 17520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:31.091975 17520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:31.092020 17520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:31.092034 17520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:31.102088 17520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40041
I20260926 11:39:31.102123 17648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40041 every 8 connection(s)
I20260926 11:39:31.103088 17520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:39:31.107173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 17520
I20260926 11:39:31.107342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:39:31.107726 392 external_mini_cluster.cc:1428] Setting key 7fd855edd215d9679f8bb03a34feefef
Sep 26 11:39:31 dist-test-slave-4pw5 krb5kdc[17413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:39:31.122488 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.105209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39411 (local address 127.0.98.62:38381)
0926 11:39:31.105380 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.105385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.106438 (+ 1053us) server_negotiation.cc:407] Connection header received
0926 11:39:31.108445 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.108450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.108541 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.108626 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:31.110365 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.111502 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.113618 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.113863 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.113968 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.117611 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:31.117616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:31.117618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:31.117652 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:31.119553 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:31.120431 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:31.120435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:31.120437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:31.120548 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:31.120997 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:31.121041 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:31.121043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:31.121305 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:31.121437 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.122104 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.122286 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:39:31.123453 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.105635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38381 (local address 127.0.98.1:39411)
0926 11:39:31.106247 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.106297 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:31.108111 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:31.108881 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:31.108891 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:31.109423 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:31.110190 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.110209 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.111742 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.111748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.113438 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.113457 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.114054 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.114061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.114334 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.115187 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:31.115218 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:31.117328 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:31.119790 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:31.119799 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:31.119821 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:31.120288 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:31.120660 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:31.120663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:31.120666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:31.120863 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:31.121436 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:31.121446 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:31.121815 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 11:39:31.122055 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 11:39:31.122465 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":136,"threads_started":1}
I20260926 11:39:31.125051 17649 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38381
I20260926 11:39:31.125353 17649 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:31.126039 17649 heartbeater.cc:511] Master 127.0.98.62:38381 requested a full tablet report, sending...
I20260926 11:39:31.128156 17458 ts_manager.cc:194] Registered new tserver with Master: 9b48b91aacc7495fb51ecefc46f38e83 (127.0.98.1:40041)
I20260926 11:39:31.129506 17458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:39:31.144801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38381
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:31.287431 17654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:31.287883 17654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:31.287971 17654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:31.292300 17654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:31.292395 17654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:31.292490 17654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:39:31.298290 17654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:31.299492 17654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:31.300575 17654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:31.310949 17659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:31.310949 17662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:31.310949 17660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:31.311705 17654 server_base.cc:1034] running on GCE node
I20260926 11:39:31.312160 17654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:31.312903 17654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:31.314109 17654 hybrid_clock.cc:648] HybridClock initialized: now 1790422771314066 us; error 59 us; skew 500 ppm
Sep 26 11:39:31 dist-test-slave-4pw5 krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:31.317323 17654 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:39:31.318408 17654 webserver.cc:460] Webserver started at http://127.0.98.2:40509/ using document root <none> and password file <none>
I20260926 11:39:31.319087 17654 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:31.319195 17654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:31.319458 17654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:31.321805 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "7f8a7cc7fb2d41fc92612352089ab77c"
format_stamp: "Formatted at 2026-09-26 11:39:31 on dist-test-slave-4pw5"
server_key: "1586846065bc1b359e5e5584914c671b"
server_key_iv: "d2bf5da797565f44e426df839a1fa4d1"
server_key_version: "encryptionkey@0"
I20260926 11:39:31.322432 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7f8a7cc7fb2d41fc92612352089ab77c"
format_stamp: "Formatted at 2026-09-26 11:39:31 on dist-test-slave-4pw5"
server_key: "1586846065bc1b359e5e5584914c671b"
server_key_iv: "d2bf5da797565f44e426df839a1fa4d1"
server_key_version: "encryptionkey@0"
I20260926 11:39:31.326579 17654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 11:39:31.329427 17669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:31.330862 17654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:39:31.331041 17654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "7f8a7cc7fb2d41fc92612352089ab77c"
format_stamp: "Formatted at 2026-09-26 11:39:31 on dist-test-slave-4pw5"
server_key: "1586846065bc1b359e5e5584914c671b"
server_key_iv: "d2bf5da797565f44e426df839a1fa4d1"
server_key_version: "encryptionkey@0"
I20260926 11:39:31.331182 17654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:31.346899 17654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:31.350476 17654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:31.350713 17654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:31.351346 17654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:31.352312 17654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:31.352387 17654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:31.352460 17654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:31.352505 17654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:31.362613 17654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44563
I20260926 11:39:31.362666 17782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44563 every 8 connection(s)
I20260926 11:39:31.363700 17654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:39:31.373037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 17654
I20260926 11:39:31.373153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:39:31.373499 392 external_mini_cluster.cc:1428] Setting key 3facae4a4f96311fb4747faebb664d31
Sep 26 11:39:31 dist-test-slave-4pw5 krb5kdc[17413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:39:31.378647 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.365840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38641 (local address 127.0.98.62:38381)
0926 11:39:31.365983 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.365987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.367060 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 11:39:31.368201 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.368205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.368257 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.368337 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:31.369544 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.370076 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.371384 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.371792 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.371894 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.374744 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:31.374749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:31.374751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:31.374807 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:31.376486 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:31.377048 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:31.377052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:31.377054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:31.377142 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:31.377499 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:31.377504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:31.377506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:31.377712 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:31.377821 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.378357 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.378484 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:39:31.379678 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.366153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38381 (local address 127.0.98.2:38641)
0926 11:39:31.366888 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.366933 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:31.367946 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:31.368508 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:31.368515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:31.368910 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:31.369399 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.369413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.370222 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.370225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.371228 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.371239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.371966 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.371969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.372147 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.372835 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:31.372853 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:31.374542 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:31.376621 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:31.376628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:31.376647 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:31.376930 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:31.377238 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:31.377243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:31.377245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:31.377400 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:31.377823 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:31.377833 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:31.378106 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 11:39:31.378314 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 11:39:31.378660 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":124,"threads_started":1}
I20260926 11:39:31.380950 17783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38381
I20260926 11:39:31.381219 17783 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:31.381728 17783 heartbeater.cc:511] Master 127.0.98.62:38381 requested a full tablet report, sending...
I20260926 11:39:31.382903 17458 ts_manager.cc:194] Registered new tserver with Master: 7f8a7cc7fb2d41fc92612352089ab77c (127.0.98.2:44563)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:39:31.383762 17458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:39:31.404296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38381
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:31.525568 17788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:31.525895 17788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:31.525954 17788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:31.530839 17788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:31.530915 17788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:31.530987 17788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:39:31.536733 17788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:31.537853 17788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:31.538873 17788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:39:31.549503 17788 server_base.cc:1034] running on GCE node
W20260926 11:39:31.549461 17796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:31.549458 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:31.549468 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:31.550220 17788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:31.550966 17788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:31.552160 17788 hybrid_clock.cc:648] HybridClock initialized: now 1790422771552143 us; error 37 us; skew 500 ppm
Sep 26 11:39:31 dist-test-slave-4pw5 krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:31.555260 17788 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:39:31.556305 17788 webserver.cc:460] Webserver started at http://127.0.98.3:39875/ using document root <none> and password file <none>
I20260926 11:39:31.556947 17788 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:31.557047 17788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:31.557296 17788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:31.559434 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "a13aeff3e5d84757a2c704a460ee5f26"
format_stamp: "Formatted at 2026-09-26 11:39:31 on dist-test-slave-4pw5"
server_key: "35e83cee8d775e9352017121c13bbfc0"
server_key_iv: "a89cc60e0e733d55de5762d7117e471b"
server_key_version: "encryptionkey@0"
I20260926 11:39:31.560143 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a13aeff3e5d84757a2c704a460ee5f26"
format_stamp: "Formatted at 2026-09-26 11:39:31 on dist-test-slave-4pw5"
server_key: "35e83cee8d775e9352017121c13bbfc0"
server_key_iv: "a89cc60e0e733d55de5762d7117e471b"
server_key_version: "encryptionkey@0"
I20260926 11:39:31.564240 17788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:39:31.567049 17803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:31.568615 17788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 11:39:31.568768 17788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "a13aeff3e5d84757a2c704a460ee5f26"
format_stamp: "Formatted at 2026-09-26 11:39:31 on dist-test-slave-4pw5"
server_key: "35e83cee8d775e9352017121c13bbfc0"
server_key_iv: "a89cc60e0e733d55de5762d7117e471b"
server_key_version: "encryptionkey@0"
I20260926 11:39:31.568895 17788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:31.585270 17788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:31.589224 17788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:31.589483 17788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:31.590166 17788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:31.591125 17788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:31.591256 17788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:31.591379 17788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:31.591432 17788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:31.600868 17788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37453
I20260926 11:39:31.600885 17916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37453 every 8 connection(s)
I20260926 11:39:31.601820 17788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:39:31.602028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 17788
I20260926 11:39:31.602147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:39:31.602499 392 external_mini_cluster.cc:1428] Setting key 1fc216c4a75d74b9782b5b0beb1195ea
Sep 26 11:39:31 dist-test-slave-4pw5 krb5kdc[17413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:39:31.616739 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.604200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57017 (local address 127.0.98.62:38381)
0926 11:39:31.604319 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.604322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.605238 (+ 916us) server_negotiation.cc:407] Connection header received
0926 11:39:31.606197 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.606199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.606247 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.606317 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:31.607563 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.608198 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.609499 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.609815 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.609902 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.612595 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:31.612598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:31.612600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:31.612663 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:31.614168 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:31.614769 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:31.614773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:31.614775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:31.614883 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:31.615245 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:31.615248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:31.615249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:31.615492 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:31.615669 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.616379 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.616530 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 11:39:31.617802 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.604595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38381 (local address 127.0.98.3:57017)
0926 11:39:31.605070 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.605114 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:31.605981 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:31.606451 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:31.606458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:31.606907 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:31.607406 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.607422 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.608330 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.608333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.609375 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.609384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.609981 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.609984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.610157 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.610861 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:31.610889 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:31.612406 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:31.614285 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:31.614290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:31.614304 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:31.614563 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:31.614973 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:31.614975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:31.614976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:31.615133 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:31.615698 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:31.615704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:31.615971 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 11:39:31.616336 (+ 365us) client_negotiation.cc:240] Negotiation successful
0926 11:39:31.616720 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260926 11:39:31.619065 17917 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38381
I20260926 11:39:31.619357 17917 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:31.619935 17917 heartbeater.cc:511] Master 127.0.98.62:38381 requested a full tablet report, sending...
I20260926 11:39:31.621125 17458 ts_manager.cc:194] Registered new tserver with Master: a13aeff3e5d84757a2c704a460ee5f26 (127.0.98.3:37453)
I20260926 11:39:31.621729 17458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57017
I20260926 11:39:31.630942 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:39:31.642118 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.633606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.0.98.62:38381)
0926 11:39:31.633806 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.633811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.633965 (+ 154us) server_negotiation.cc:407] Connection header received
0926 11:39:31.634218 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.634227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.634275 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.634368 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:31.635214 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.635887 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.636849 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.637098 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.637223 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.638801 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:31.638804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:31.638806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:31.638835 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:31.640330 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:31.640811 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:31.640815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:31.640817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:31.640903 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:31.641196 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:31.641199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:31.641200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:31.641442 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:31.641597 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.641832 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.641947 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 11:39:31.645560 17458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:39:31.648721 17458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:39:31.664841 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.658366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.0.98.2:44563)
0926 11:39:31.658848 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.658862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.659489 (+ 627us) server_negotiation.cc:407] Connection header received
0926 11:39:31.659605 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.659611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.659832 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.659993 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:31.660554 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.661467 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.663182 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.664073 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.664198 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.664382 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.664530 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.664647 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":116,"threads_started":1}
I20260926 11:39:31.664930 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.658137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44563 (local address 127.0.0.1:46654)
0926 11:39:31.659143 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.659202 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:31.659340 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:31.660195 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:31.660197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:31.660213 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:31.660384 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.660391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.661619 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.661622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.663051 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.663060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.664194 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.664198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.664327 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.664373 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 11:39:31.664497 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 11:39:31.664747 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":29,"spinlock_wait_cycles":4864,"thread_start_us":95,"threads_started":1}
I20260926 11:39:31.665227 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.658385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37453 (local address 127.0.0.1:42678)
0926 11:39:31.659140 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.659200 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:31.659339 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:31.660035 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:31.660041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:31.660068 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:31.660294 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.660303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.661390 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.661394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.663259 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.663270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.664460 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.664464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.664579 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.664602 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:39:31.664708 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 11:39:31.664789 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":27,"spinlock_wait_cycles":16000,"thread_start_us":113,"threads_started":1}
I20260926 11:39:31.665963 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.659029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.0.98.3:37453)
0926 11:39:31.659512 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.659525 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.659543 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:39:31.659634 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.659638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.659832 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.659994 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:31.660429 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.661249 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.663397 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.664220 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.664315 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.664424 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.665673 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.665808 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":117,"threads_started":1}
I20260926 11:39:31.666153 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.658142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40041 (local address 127.0.0.1:33848)
0926 11:39:31.659652 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.659669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:31.659767 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:31.661384 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:31.661387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:31.661404 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:31.661475 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.661479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.662873 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.662876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.663950 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.663959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.665706 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.665710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.665814 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.665839 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 11:39:31.665956 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 11:39:31.666034 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"mutex_wait_us":648,"thread_start_us":124,"threads_started":1}
I20260926 11:39:31.666626 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.658391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.0.98.1:40041)
0926 11:39:31.659876 (+ 1485us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.659890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.659956 (+ 66us) server_negotiation.cc:407] Connection header received
0926 11:39:31.660060 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.660065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.660265 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.660432 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:31.661617 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.662748 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.664081 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.665048 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.665472 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.665628 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.666379 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.666485 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1358,"thread_start_us":101,"threads_started":1}
I20260926 11:39:31.668499 17717 tablet_service.cc:1467] Processing CreateTablet for tablet a0f0b79b1ae6435dacc82a12fb5ec130 (DEFAULT_TABLE table=test-table [id=a1ba0bc997424f329f63632b7e75884a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:31.668756 17851 tablet_service.cc:1467] Processing CreateTablet for tablet a0f0b79b1ae6435dacc82a12fb5ec130 (DEFAULT_TABLE table=test-table [id=a1ba0bc997424f329f63632b7e75884a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:31.668890 17583 tablet_service.cc:1467] Processing CreateTablet for tablet a0f0b79b1ae6435dacc82a12fb5ec130 (DEFAULT_TABLE table=test-table [id=a1ba0bc997424f329f63632b7e75884a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:31.669726 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f0b79b1ae6435dacc82a12fb5ec130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:31.669726 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f0b79b1ae6435dacc82a12fb5ec130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:31.669871 17583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f0b79b1ae6435dacc82a12fb5ec130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:31.676921 17934 tablet_bootstrap.cc:492] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83: Bootstrap starting.
I20260926 11:39:31.677951 17935 tablet_bootstrap.cc:492] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: Bootstrap starting.
I20260926 11:39:31.678416 17936 tablet_bootstrap.cc:492] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: Bootstrap starting.
I20260926 11:39:31.679342 17934 tablet_bootstrap.cc:654] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:31.680281 17934 log.cc:824] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:31.680399 17936 tablet_bootstrap.cc:654] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:31.681656 17936 log.cc:824] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:31.681710 17935 tablet_bootstrap.cc:654] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:31.682557 17934 tablet_bootstrap.cc:492] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83: No bootstrap required, opened a new log
I20260926 11:39:31.682767 17934 ts_tablet_manager.cc:1397] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 11:39:31.682808 17935 log.cc:824] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:31.683706 17936 tablet_bootstrap.cc:492] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: No bootstrap required, opened a new log
I20260926 11:39:31.683887 17936 ts_tablet_manager.cc:1397] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:39:31.684737 17935 tablet_bootstrap.cc:492] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: No bootstrap required, opened a new log
I20260926 11:39:31.684926 17935 ts_tablet_manager.cc:1397] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 11:39:31.686556 17934 raft_consensus.cc:348] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260926 11:39:31.686817 17934 raft_consensus.cc:374] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:31.686897 17934 raft_consensus.cc:729] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b48b91aacc7495fb51ecefc46f38e83, State: Initialized, Role: FOLLOWER
I20260926 11:39:31.687686 17934 consensus_queue.cc:260] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260926 11:39:31.687722 17936 raft_consensus.cc:348] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260926 11:39:31.688053 17936 raft_consensus.cc:374] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:31.688133 17936 raft_consensus.cc:729] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8a7cc7fb2d41fc92612352089ab77c, State: Initialized, Role: FOLLOWER
I20260926 11:39:31.688501 17649 heartbeater.cc:503] Master 127.0.98.62:38381 was elected leader, sending a full tablet report...
I20260926 11:39:31.688709 17936 consensus_queue.cc:260] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260926 11:39:31.688846 17934 ts_tablet_manager.cc:1428] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:39:31.688679 17935 raft_consensus.cc:348] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260926 11:39:31.688944 17935 raft_consensus.cc:374] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:31.689040 17935 raft_consensus.cc:729] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13aeff3e5d84757a2c704a460ee5f26, State: Initialized, Role: FOLLOWER
I20260926 11:39:31.689477 17935 consensus_queue.cc:260] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260926 11:39:31.689808 17936 ts_tablet_manager.cc:1428] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:39:31.690136 17917 heartbeater.cc:503] Master 127.0.98.62:38381 was elected leader, sending a full tablet report...
I20260926 11:39:31.690168 17783 heartbeater.cc:503] Master 127.0.98.62:38381 was elected leader, sending a full tablet report...
I20260926 11:39:31.690382 17935 ts_tablet_manager.cc:1428] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:39:31.792346 17942 raft_consensus.cc:482] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:31.792554 17942 raft_consensus.cc:504] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260926 11:39:31.793733 17942 leader_election.cc:288] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8a7cc7fb2d41fc92612352089ab77c (127.0.98.2:44563), 9b48b91aacc7495fb51ecefc46f38e83 (127.0.98.1:40041)
I20260926 11:39:31.797427 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.793963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44563 (local address 127.0.98.3:53659)
0926 11:39:31.794136 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.794153 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:31.794279 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:31.794576 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:31.794579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:31.794595 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:31.794733 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.794738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.795507 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.795512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.796313 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.796322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.796988 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.796991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.797105 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.797124 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 11:39:31.797210 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 11:39:31.797289 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"spinlock_wait_cycles":35072}
I20260926 11:39:31.797427 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.794092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53659 (local address 127.0.98.2:44563)
0926 11:39:31.794238 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.794242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.794262 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:39:31.794407 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.794410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.794452 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.794559 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:31.794844 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.795392 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.796443 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.796861 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.796980 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.797046 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.797226 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.797303 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:39:31.797820 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.794259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40093 (local address 127.0.98.1:40041)
0926 11:39:31.794409 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.794413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.794568 (+ 155us) server_negotiation.cc:407] Connection header received
0926 11:39:31.794764 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.794766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.794806 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.794887 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:31.795158 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.795736 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.796785 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.797205 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.797329 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.797388 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.797617 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.797706 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:39:31.798158 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.794003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40041 (local address 127.0.98.3:40093)
0926 11:39:31.794468 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.794480 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:31.794629 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:31.794919 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:31.794922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:31.794937 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:31.795052 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.795057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.795847 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.795852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.796654 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:31.796662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.797335 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.797339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:31.797437 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.797465 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 11:39:31.797960 (+ 495us) client_negotiation.cc:240] Negotiation successful
0926 11:39:31.798040 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"spinlock_wait_cycles":768,"thread_start_us":128,"threads_started":1}
I20260926 11:39:31.798183 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f0b79b1ae6435dacc82a12fb5ec130" candidate_uuid: "a13aeff3e5d84757a2c704a460ee5f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" is_pre_election: true
I20260926 11:39:31.798574 17737 raft_consensus.cc:2454] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13aeff3e5d84757a2c704a460ee5f26 in term 0.
I20260926 11:39:31.798887 17603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f0b79b1ae6435dacc82a12fb5ec130" candidate_uuid: "a13aeff3e5d84757a2c704a460ee5f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b48b91aacc7495fb51ecefc46f38e83" is_pre_election: true
I20260926 11:39:31.799150 17807 leader_election.cc:302] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8a7cc7fb2d41fc92612352089ab77c, a13aeff3e5d84757a2c704a460ee5f26; no voters:
I20260926 11:39:31.799172 17603 raft_consensus.cc:2454] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13aeff3e5d84757a2c704a460ee5f26 in term 0.
I20260926 11:39:31.799428 17942 raft_consensus.cc:2781] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:39:31.799536 17942 raft_consensus.cc:482] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:39:31.799610 17942 raft_consensus.cc:3037] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:31.800676 17942 raft_consensus.cc:504] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260926 11:39:31.801079 17942 leader_election.cc:288] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [CANDIDATE]: Term 1 election: Requested vote from peers 7f8a7cc7fb2d41fc92612352089ab77c (127.0.98.2:44563), 9b48b91aacc7495fb51ecefc46f38e83 (127.0.98.1:40041)
I20260926 11:39:31.801460 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f0b79b1ae6435dacc82a12fb5ec130" candidate_uuid: "a13aeff3e5d84757a2c704a460ee5f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8a7cc7fb2d41fc92612352089ab77c"
I20260926 11:39:31.801625 17737 raft_consensus.cc:3037] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:31.801584 17603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f0b79b1ae6435dacc82a12fb5ec130" candidate_uuid: "a13aeff3e5d84757a2c704a460ee5f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b48b91aacc7495fb51ecefc46f38e83"
I20260926 11:39:31.801769 17603 raft_consensus.cc:3037] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:31.802731 17737 raft_consensus.cc:2454] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13aeff3e5d84757a2c704a460ee5f26 in term 1.
I20260926 11:39:31.803022 17603 raft_consensus.cc:2454] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13aeff3e5d84757a2c704a460ee5f26 in term 1.
I20260926 11:39:31.803041 17807 leader_election.cc:302] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8a7cc7fb2d41fc92612352089ab77c, a13aeff3e5d84757a2c704a460ee5f26; no voters:
I20260926 11:39:31.803198 17942 raft_consensus.cc:2781] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:31.803467 17942 raft_consensus.cc:686] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 1 LEADER]: Becoming Leader. State: Replica: a13aeff3e5d84757a2c704a460ee5f26, State: Running, Role: LEADER
I20260926 11:39:31.803851 17942 consensus_queue.cc:237] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } }
I20260926 11:39:31.807036 17457 catalog_manager.cc:5495] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to a13aeff3e5d84757a2c704a460ee5f26 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a13aeff3e5d84757a2c704a460ee5f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a13aeff3e5d84757a2c704a460ee5f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37453 } health_report { overall_health: HEALTHY } } }
I20260926 11:39:31.818715 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.815005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.0.98.3:37453)
0926 11:39:31.815130 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:31.815134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:31.815147 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:39:31.815282 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:31.815284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:31.815341 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:31.815432 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:39:31.815694 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.816287 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.817266 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:31.817479 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:31.817564 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:31.817865 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:39:31.818350 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:39:31.818444 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:31.818548 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 11:39:31.818603 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:39:31.824800 17737 raft_consensus.cc:1261] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [term 1 FOLLOWER]: Refusing update from remote peer a13aeff3e5d84757a2c704a460ee5f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:31.824800 17603 raft_consensus.cc:1261] T a0f0b79b1ae6435dacc82a12fb5ec130 P 9b48b91aacc7495fb51ecefc46f38e83 [term 1 FOLLOWER]: Refusing update from remote peer a13aeff3e5d84757a2c704a460ee5f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:31.825565 17944 consensus_queue.cc:1035] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b48b91aacc7495fb51ecefc46f38e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:31.825784 17942 consensus_queue.cc:1035] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8a7cc7fb2d41fc92612352089ab77c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:31.833673 17947 mvcc.cc:205] Tried to move back new op lower bound from 7333571673386483712 to 7333571673311277056. Current Snapshot: MvccSnapshot[applied={T|T < 7333571673386483712}]
I20260926 11:39:31.834066 17949 mvcc.cc:205] Tried to move back new op lower bound from 7333571673386483712 to 7333571673311277056. Current Snapshot: MvccSnapshot[applied={T|T < 7333571673386483712}]
I20260926 11:39:31.844837 17458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:31.845048 17458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:31.848053 17458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7246937463e414db83da32a1a08bef2: Sending DeleteTablet for 3 replicas of tablet a0f0b79b1ae6435dacc82a12fb5ec130
I20260926 11:39:31.849009 17583 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f0b79b1ae6435dacc82a12fb5ec130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33848
I20260926 11:39:31.849197 17717 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f0b79b1ae6435dacc82a12fb5ec130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46654
I20260926 11:39:31.849784 17958 tablet_replica.cc:330] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: stopping tablet replica
I20260926 11:39:31.849834 17851 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f0b79b1ae6435dacc82a12fb5ec130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42678
I20260926 11:39:31.850198 17958 raft_consensus.cc:2229] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:31.850302 17959 tablet_replica.cc:330] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: stopping tablet replica
I20260926 11:39:31.850327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 17520
I20260926 11:39:31.851781 17959 raft_consensus.cc:2229] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:39:31.852329 17959 raft_consensus.cc:2258] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:31.852800 17958 raft_consensus.cc:2258] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:31.854517 17959 ts_tablet_manager.cc:1905] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:31.854790 17958 ts_tablet_manager.cc:1905] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:31.857524 17959 ts_tablet_manager.cc:1918] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:31.857743 17959 log.cc:1196] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/a0f0b79b1ae6435dacc82a12fb5ec130
I20260926 11:39:31.858171 17959 ts_tablet_manager.cc:1939] T a0f0b79b1ae6435dacc82a12fb5ec130 P a13aeff3e5d84757a2c704a460ee5f26: Deleting consensus metadata
I20260926 11:39:31.858371 17958 ts_tablet_manager.cc:1918] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:31.858557 17958 log.cc:1196] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/a0f0b79b1ae6435dacc82a12fb5ec130
W20260926 11:39:31.858879 17443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40041 (error 108)
I20260926 11:39:31.858963 17958 ts_tablet_manager.cc:1939] T a0f0b79b1ae6435dacc82a12fb5ec130 P 7f8a7cc7fb2d41fc92612352089ab77c: Deleting consensus metadata
I20260926 11:39:31.859643 17442 catalog_manager.cc:4841] TS a13aeff3e5d84757a2c704a460ee5f26 (127.0.98.3:37453): tablet a0f0b79b1ae6435dacc82a12fb5ec130 (table test-table [id=a1ba0bc997424f329f63632b7e75884a]) successfully deleted
I20260926 11:39:31.860235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 17654
I20260926 11:39:31.860635 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.860257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40041 (local address 127.0.0.1:33858)
0926 11:39:31.860438 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.860526 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 11:39:31.861197 17443 catalog_manager.cc:4584] TS 9b48b91aacc7495fb51ecefc46f38e83 (127.0.98.1:40041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0f0b79b1ae6435dacc82a12fb5ec130: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40041: connect: Connection refused (error 111)
I20260926 11:39:31.867221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 17788
I20260926 11:39:31.867482 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:31.867108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44563 (local address 127.0.0.1:46666)
0926 11:39:31.867287 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:31.867366 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 11:39:31.872752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 17426
2026-09-26T11:39:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2499 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T11:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:31Z Disabled control of system clock
I20260926 11:39:31.903345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39601
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:39:32.020260 17964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:32.020577 17964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:32.020617 17964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:32.024921 17964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:32.025005 17964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:32.025022 17964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:32.025039 17964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:32.025055 17964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:32.030366 17964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:32.031672 17964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:32.033128 17964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:32.042367 17969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:32.042444 17972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:32.042444 17970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:32.042647 17964 server_base.cc:1034] running on GCE node
I20260926 11:39:32.043267 17964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:32.044617 17964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:32.045993 17964 hybrid_clock.cc:648] HybridClock initialized: now 1790422772045965 us; error 53 us; skew 500 ppm
I20260926 11:39:32.048080 17964 webserver.cc:460] Webserver started at http://127.0.98.62:46123/ using document root <none> and password file <none>
I20260926 11:39:32.048696 17964 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:32.048794 17964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:32.049037 17964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:32.051254 17964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "48e63f626b544e0897e752d44f60d0cc"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "e3e8a04cb6896d7578aeff3641711a72"
server_key_iv: "77b160c2cdea44944f03e664958a2967"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.051851 17964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "48e63f626b544e0897e752d44f60d0cc"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "e3e8a04cb6896d7578aeff3641711a72"
server_key_iv: "77b160c2cdea44944f03e664958a2967"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.056255 17964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:39:32.058951 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.060460 17964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:32.060637 17964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "48e63f626b544e0897e752d44f60d0cc"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "e3e8a04cb6896d7578aeff3641711a72"
server_key_iv: "77b160c2cdea44944f03e664958a2967"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.060779 17964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:32.078979 17964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:32.079862 17964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:32.080078 17964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:32.087713 17964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39601
I20260926 11:39:32.087745 18030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39601 every 8 connection(s)
I20260926 11:39:32.088749 17964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:32.090023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 17964
I20260926 11:39:32.090155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:32.090468 392 external_mini_cluster.cc:1428] Setting key c9c28a669ca3475f5284d51c6b5b3058
I20260926 11:39:32.092935 18031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:32.099021 18031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc: Bootstrap starting.
I20260926 11:39:32.100793 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.091984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.0.98.62:39601)
0926 11:39:32.092639 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.092655 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.092793 (+ 138us) server_negotiation.cc:407] Connection header received
0926 11:39:32.093763 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.093781 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.094199 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.094583 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:32.095041 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.096173 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.097141 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.097650 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.097843 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.098869 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:32.098873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:32.098886 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:32.099364 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:32.099425 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:32.099447 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:32.099565 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.099767 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.100041 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":140,"threads_started":1}
I20260926 11:39:32.101588 18031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:32.102557 18031 log.cc:824] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:32.105033 18031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc: No bootstrap required, opened a new log
I20260926 11:39:32.109421 18031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e63f626b544e0897e752d44f60d0cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260926 11:39:32.109716 18031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:32.109756 18031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e63f626b544e0897e752d44f60d0cc, State: Initialized, Role: FOLLOWER
I20260926 11:39:32.110200 18031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e63f626b544e0897e752d44f60d0cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260926 11:39:32.110333 18031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:32.110383 18031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:32.110458 18031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:32.111562 18031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e63f626b544e0897e752d44f60d0cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260926 11:39:32.112071 18031 leader_election.cc:302] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48e63f626b544e0897e752d44f60d0cc; no voters:
I20260926 11:39:32.112455 18031 leader_election.cc:288] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:32.112597 18036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:32.112833 18036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [term 1 LEADER]: Becoming Leader. State: Replica: 48e63f626b544e0897e752d44f60d0cc, State: Running, Role: LEADER
I20260926 11:39:32.113200 18036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e63f626b544e0897e752d44f60d0cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260926 11:39:32.113793 18031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:32.114871 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48e63f626b544e0897e752d44f60d0cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e63f626b544e0897e752d44f60d0cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } } }
I20260926 11:39:32.115005 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:32.115388 18045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:32.115391 18038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48e63f626b544e0897e752d44f60d0cc. Latest consensus state: current_term: 1 leader_uuid: "48e63f626b544e0897e752d44f60d0cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e63f626b544e0897e752d44f60d0cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } } }
I20260926 11:39:32.116236 18038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:32.118894 18045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:32.126221 18045 catalog_manager.cc:1347] Generated new cluster ID: a8fd82afbbd64b1a80c54176f111d830
I20260926 11:39:32.126320 18045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:32.171140 18045 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:32.172343 18045 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:32.178349 18045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc: Generated new TSK 0
I20260926 11:39:32.179210 18045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:32.182441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--builtin_ntp_servers=127.0.98.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:39:32.317662 18055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:32.318013 18055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:32.318056 18055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:32.322551 18055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:32.322625 18055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:32.322690 18055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:39:32.327814 18055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:32.329229 18055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:32.330884 18055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:32.341393 18063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:32.341411 18061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:32.341475 18060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:32.341624 18055 server_base.cc:1034] running on GCE node
I20260926 11:39:32.342239 18055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:32.343012 18055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:32.344237 18055 hybrid_clock.cc:648] HybridClock initialized: now 1790422772344218 us; error 38 us; skew 500 ppm
I20260926 11:39:32.346313 18055 webserver.cc:460] Webserver started at http://127.0.98.1:38845/ using document root <none> and password file <none>
I20260926 11:39:32.346951 18055 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:32.347054 18055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:32.347302 18055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:32.349913 18055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "58f1daee2cd14c2fbadeae687ad1c7f3"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "eb200e804a63aa666d959fda96b3f9af"
server_key_iv: "3af4d6bed9f96c772dd6850e319b445f"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.350560 18055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "58f1daee2cd14c2fbadeae687ad1c7f3"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "eb200e804a63aa666d959fda96b3f9af"
server_key_iv: "3af4d6bed9f96c772dd6850e319b445f"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.354589 18055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:39:32.357404 18069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.358788 18055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:39:32.358939 18055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "58f1daee2cd14c2fbadeae687ad1c7f3"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "eb200e804a63aa666d959fda96b3f9af"
server_key_iv: "3af4d6bed9f96c772dd6850e319b445f"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.359071 18055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:32.382942 18055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:32.383741 18055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:32.383951 18055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:32.384565 18055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:32.385527 18055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:32.385599 18055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.385671 18055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:32.385737 18055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.395457 18055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39403
I20260926 11:39:32.395550 18182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39403 every 8 connection(s)
I20260926 11:39:32.396656 18055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:39:32.400705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 18055
I20260926 11:39:32.400811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:39:32.401089 392 external_mini_cluster.cc:1428] Setting key c10a24aa6049804c47bfb5f0bc99d385
I20260926 11:39:32.403437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--builtin_ntp_servers=127.0.98.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:39:32.407429 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.398648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35195 (local address 127.0.98.62:39601)
0926 11:39:32.398817 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.398822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.399949 (+ 1127us) server_negotiation.cc:407] Connection header received
0926 11:39:32.401482 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.401489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.401555 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.401635 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:32.402826 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.403678 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.405363 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.405579 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.405685 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.406896 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:32.406899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:32.406901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:32.406914 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:32.406934 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:32.406942 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:32.407017 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.407191 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.407279 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:39:32.408344 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.399097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.1:35195)
0926 11:39:32.399805 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:32.399844 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:32.401264 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:32.401834 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:32.401841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:32.401956 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:32.402653 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.402676 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.403838 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.403844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.405218 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.405228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.405755 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.405760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.405924 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.406584 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:39:32.406586 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:32.406640 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:39:32.406717 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:32.407032 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:32.407042 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:32.407054 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 11:39:32.407193 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 11:39:32.407497 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260926 11:39:32.409686 18183 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260926 11:39:32.409950 18183 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:32.410511 18183 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260926 11:39:32.412366 17995 ts_manager.cc:194] Registered new tserver with Master: 58f1daee2cd14c2fbadeae687ad1c7f3 (127.0.98.1:39403)
I20260926 11:39:32.414319 17995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35195
W20260926 11:39:32.522207 18186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:32.522498 18186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:32.522537 18186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:32.526834 18186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:32.526964 18186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:32.527084 18186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:39:32.532681 18186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:32.533874 18186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:32.535306 18186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:32.546232 18194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:32.546232 18192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:32.546293 18191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:32.546478 18186 server_base.cc:1034] running on GCE node
I20260926 11:39:32.547082 18186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:32.547902 18186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:32.549153 18186 hybrid_clock.cc:648] HybridClock initialized: now 1790422772549124 us; error 38 us; skew 500 ppm
I20260926 11:39:32.551306 18186 webserver.cc:460] Webserver started at http://127.0.98.2:39703/ using document root <none> and password file <none>
I20260926 11:39:32.552034 18186 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:32.552139 18186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:32.552382 18186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:32.554653 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "c222466b5f7a491b8eb69ab3b3afbcaf"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "e9ad8ba7abf57714b3dcbafcd6f03040"
server_key_iv: "8a035acd1afeafdf64939be840e9d79e"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.555289 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c222466b5f7a491b8eb69ab3b3afbcaf"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "e9ad8ba7abf57714b3dcbafcd6f03040"
server_key_iv: "8a035acd1afeafdf64939be840e9d79e"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.559760 18186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:32.562646 18200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.564288 18186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:32.564430 18186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "c222466b5f7a491b8eb69ab3b3afbcaf"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "e9ad8ba7abf57714b3dcbafcd6f03040"
server_key_iv: "8a035acd1afeafdf64939be840e9d79e"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.564566 18186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:32.575344 18186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:32.576030 18186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:32.576192 18186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:32.576709 18186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:32.577703 18186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:32.577750 18186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.577790 18186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:32.577805 18186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.586781 18186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46695
I20260926 11:39:32.586800 18313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46695 every 8 connection(s)
I20260926 11:39:32.587826 18186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:39:32.590902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 18186
I20260926 11:39:32.591048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:39:32.591416 392 external_mini_cluster.cc:1428] Setting key c387a18d81df5d3e99f690d6fcda1a6a
I20260926 11:39:32.593945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--builtin_ntp_servers=127.0.98.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:39:32.600960 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.589980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47865 (local address 127.0.98.62:39601)
0926 11:39:32.590177 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.590182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.591508 (+ 1326us) server_negotiation.cc:407] Connection header received
0926 11:39:32.593586 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.593589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.593645 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.593725 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:32.595310 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.595998 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.597973 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.598330 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.598431 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.600349 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:32.600354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:32.600357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:32.600380 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:32.600410 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:32.600421 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:32.600515 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.600772 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.600836 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 11:39:32.602514 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.590376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.2:47865)
0926 11:39:32.591317 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:32.591367 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:32.593351 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:32.593920 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:32.593927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:32.594099 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:32.595078 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.595104 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.596160 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.596163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.597808 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.597825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.598551 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.598558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.598856 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.599662 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:39:32.599667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:32.599877 (+ 210us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:39:32.600027 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:32.600557 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:32.600569 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:32.600581 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 11:39:32.600767 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 11:39:32.601174 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":193,"threads_started":1}
I20260926 11:39:32.604023 18314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260926 11:39:32.604482 18314 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:32.605337 18314 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260926 11:39:32.606810 17995 ts_manager.cc:194] Registered new tserver with Master: c222466b5f7a491b8eb69ab3b3afbcaf (127.0.98.2:46695)
I20260926 11:39:32.607825 17995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47865
W20260926 11:39:32.720327 18317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:32.720723 18317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:32.720772 18317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:32.724946 18317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:32.725014 18317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:32.725078 18317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:39:32.730172 18317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:32.731402 18317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:32.732884 18317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:32.744334 18322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:32.744498 18325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:32.744498 18323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:32.744781 18317 server_base.cc:1034] running on GCE node
I20260926 11:39:32.745311 18317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:32.746119 18317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:32.747356 18317 hybrid_clock.cc:648] HybridClock initialized: now 1790422772747308 us; error 61 us; skew 500 ppm
I20260926 11:39:32.749641 18317 webserver.cc:460] Webserver started at http://127.0.98.3:33783/ using document root <none> and password file <none>
I20260926 11:39:32.750552 18317 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:32.750945 18317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:32.751286 18317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:32.753682 18317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "df04096f11aa49abb090750421d9e5cf"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "3b351c97160daaae631be3a1f3608b06"
server_key_iv: "6af49d89cea0c7553a20a20ea307e042"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.754300 18317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df04096f11aa49abb090750421d9e5cf"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "3b351c97160daaae631be3a1f3608b06"
server_key_iv: "6af49d89cea0c7553a20a20ea307e042"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.758551 18317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:39:32.761739 18331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.763281 18317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 11:39:32.763477 18317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "df04096f11aa49abb090750421d9e5cf"
format_stamp: "Formatted at 2026-09-26 11:39:32 on dist-test-slave-4pw5"
server_key: "3b351c97160daaae631be3a1f3608b06"
server_key_iv: "6af49d89cea0c7553a20a20ea307e042"
server_key_version: "encryptionkey@0"
I20260926 11:39:32.763671 18317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:32.796144 18317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:32.796986 18317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:32.797230 18317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:32.797922 18317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:32.799037 18317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:32.799124 18317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.799209 18317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:32.799256 18317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:32.809450 18317 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35309
I20260926 11:39:32.809485 18444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35309 every 8 connection(s)
I20260926 11:39:32.810508 18317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:39:32.812315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 18317
I20260926 11:39:32.812439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:39:32.812781 392 external_mini_cluster.cc:1428] Setting key 111f36bd3c2780844931c98bd94aa12c
I20260926 11:39:32.820662 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.812548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55209 (local address 127.0.98.62:39601)
0926 11:39:32.812716 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.812721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.814003 (+ 1282us) server_negotiation.cc:407] Connection header received
0926 11:39:32.815728 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.815731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.815779 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.815848 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:32.816969 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.817570 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.818747 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.818979 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.819065 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.820229 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:32.820232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:32.820234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:32.820247 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:32.820267 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:32.820275 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:32.820340 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.820476 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.820534 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:39:32.821533 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.813053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.3:55209)
0926 11:39:32.813834 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:32.813887 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:32.815463 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:32.815997 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:32.816004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:32.816240 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:32.816812 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.816827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.817717 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.817719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.818627 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.818637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.819112 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.819114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.819253 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.819957 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:39:32.819959 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:32.819999 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:39:32.820067 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:32.820352 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:32.820358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:32.820363 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:39:32.820457 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 11:39:32.820713 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":139,"threads_started":1}
I20260926 11:39:32.822705 18445 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260926 11:39:32.823005 18445 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:32.823551 18445 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260926 11:39:32.824779 17995 ts_manager.cc:194] Registered new tserver with Master: df04096f11aa49abb090750421d9e5cf (127.0.98.3:35309)
I20260926 11:39:32.825371 17995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55209
I20260926 11:39:32.827260 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:39:32.833461 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.829352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.0.98.62:39601)
0926 11:39:32.829478 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.829482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.829713 (+ 231us) server_negotiation.cc:407] Connection header received
0926 11:39:32.829790 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.829792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.829834 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.829908 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:32.830238 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.830765 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.831799 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.832012 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.832108 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.833012 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:32.833016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:32.833018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:32.833031 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:32.833051 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:32.833059 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:32.833158 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.833294 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.833347 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:39:32.836853 17995 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:39:32.840276 17995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:39:32.856117 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.849956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35309 (local address 127.0.0.1:58060)
0926 11:39:32.850950 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:32.851011 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:32.851170 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:32.851774 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:32.851781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:32.851805 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:32.851962 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.851972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.853510 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.853514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.854652 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.854662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.855505 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.855509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.855649 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.855695 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 11:39:32.855824 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 11:39:32.855946 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":117,"threads_started":1}
I20260926 11:39:32.856168 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.850147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.0.98.3:35309)
0926 11:39:32.851169 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.851187 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.851229 (+ 42us) server_negotiation.cc:407] Connection header received
0926 11:39:32.851319 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.851325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.851558 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.851785 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:32.852104 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.853379 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.854783 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.855365 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.855484 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.855705 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.855867 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.855973 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":188,"threads_started":1}
I20260926 11:39:32.857566 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.850166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46695 (local address 127.0.0.1:53980)
0926 11:39:32.851256 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:32.851270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:32.851366 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:32.852350 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:32.852353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:32.852372 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:32.852475 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.852482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.853818 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.853821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.855143 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.855155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.857079 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.857083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.857175 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.857198 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:39:32.857254 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 11:39:32.857328 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":57,"thread_start_us":93,"threads_started":1}
I20260926 11:39:32.857928 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.850819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.0.98.2:46695)
0926 11:39:32.851817 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.851831 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.851856 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:39:32.851934 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.851942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.852193 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.852346 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:32.852603 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.853693 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.855297 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.856939 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.857449 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.857620 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.857683 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.857777 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":132,"threads_started":1}
I20260926 11:39:32.857966 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.851195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.0.98.1:39403)
0926 11:39:32.851532 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.851544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.851953 (+ 409us) server_negotiation.cc:407] Connection header received
0926 11:39:32.852039 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.852047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.852226 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.852389 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:32.852776 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.853825 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.855532 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.856396 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.856532 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.856741 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.857639 (+ 898us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.857807 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":128,"threads_started":1}
I20260926 11:39:32.857566 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.850766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39403 (local address 127.0.0.1:40502)
0926 11:39:32.851770 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:32.851784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:32.851890 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:32.852553 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:32.852560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:32.852581 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:32.852657 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.852661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.853960 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.853963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.855362 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.855371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.856967 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.856970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.857062 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.857086 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:39:32.857262 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 11:39:32.857349 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":26,"thread_start_us":98,"threads_started":1}
I20260926 11:39:32.859666 18379 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 (DEFAULT_TABLE table=test-table [id=3e7a310c635243cc839ef400134e1a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:32.860874 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:32.860809 18248 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 (DEFAULT_TABLE table=test-table [id=3e7a310c635243cc839ef400134e1a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:32.860818 18117 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 (DEFAULT_TABLE table=test-table [id=3e7a310c635243cc839ef400134e1a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:32.861827 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:32.861927 18117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:32.870103 18462 tablet_bootstrap.cc:492] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3: Bootstrap starting.
I20260926 11:39:32.871942 18463 tablet_bootstrap.cc:492] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: Bootstrap starting.
I20260926 11:39:32.872373 18462 tablet_bootstrap.cc:654] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:32.872407 18464 tablet_bootstrap.cc:492] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: Bootstrap starting.
I20260926 11:39:32.873308 18462 log.cc:824] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:32.874647 18464 tablet_bootstrap.cc:654] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:32.875442 18462 tablet_bootstrap.cc:492] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3: No bootstrap required, opened a new log
I20260926 11:39:32.875463 18464 log.cc:824] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:32.875701 18462 ts_tablet_manager.cc:1397] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:39:32.875878 18463 tablet_bootstrap.cc:654] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:32.876636 18463 log.cc:824] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:32.877677 18464 tablet_bootstrap.cc:492] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: No bootstrap required, opened a new log
I20260926 11:39:32.877894 18464 ts_tablet_manager.cc:1397] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:39:32.878371 18463 tablet_bootstrap.cc:492] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: No bootstrap required, opened a new log
I20260926 11:39:32.878562 18463 ts_tablet_manager.cc:1397] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 11:39:32.880692 18462 raft_consensus.cc:348] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } }
I20260926 11:39:32.881013 18462 raft_consensus.cc:374] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:32.881078 18462 raft_consensus.cc:729] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f1daee2cd14c2fbadeae687ad1c7f3, State: Initialized, Role: FOLLOWER
I20260926 11:39:32.881599 18462 consensus_queue.cc:260] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } }
I20260926 11:39:32.882397 18183 heartbeater.cc:503] Master 127.0.98.62:39601 was elected leader, sending a full tablet report...
I20260926 11:39:32.882799 18462 ts_tablet_manager.cc:1428] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260926 11:39:32.882630 18464 raft_consensus.cc:348] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } }
I20260926 11:39:32.882902 18464 raft_consensus.cc:374] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:32.882963 18464 raft_consensus.cc:729] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df04096f11aa49abb090750421d9e5cf, State: Initialized, Role: FOLLOWER
I20260926 11:39:32.882206 18463 raft_consensus.cc:348] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } }
I20260926 11:39:32.883430 18464 consensus_queue.cc:260] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } }
I20260926 11:39:32.883540 18463 raft_consensus.cc:374] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:32.883621 18463 raft_consensus.cc:729] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c222466b5f7a491b8eb69ab3b3afbcaf, State: Initialized, Role: FOLLOWER
I20260926 11:39:32.884056 18463 consensus_queue.cc:260] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } }
I20260926 11:39:32.884132 18445 heartbeater.cc:503] Master 127.0.98.62:39601 was elected leader, sending a full tablet report...
I20260926 11:39:32.884480 18464 ts_tablet_manager.cc:1428] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 11:39:32.884930 18463 ts_tablet_manager.cc:1428] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:39:32.885267 18314 heartbeater.cc:503] Master 127.0.98.62:39601 was elected leader, sending a full tablet report...
W20260926 11:39:32.898049 18184 tablet.cc:2369] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:39:32.943833 18469 raft_consensus.cc:482] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:32.944032 18469 raft_consensus.cc:504] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } }
I20260926 11:39:32.945195 18469 leader_election.cc:288] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c222466b5f7a491b8eb69ab3b3afbcaf (127.0.98.2:46695), 58f1daee2cd14c2fbadeae687ad1c7f3 (127.0.98.1:39403)
I20260926 11:39:32.949548 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.945460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39403 (local address 127.0.98.3:43007)
0926 11:39:32.945599 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:32.945616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:32.945748 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:32.946189 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:32.946191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:32.946204 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:32.946340 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.946348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.947436 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.947440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.948415 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.948426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.949118 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.949124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.949226 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.949250 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:39:32.949322 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 11:39:32.949387 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 11:39:32.949592 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.945550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43007 (local address 127.0.98.1:39403)
0926 11:39:32.945688 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.945693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.945726 (+ 33us) server_negotiation.cc:407] Connection header received
0926 11:39:32.945895 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.945899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.945955 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.946040 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:32.946494 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.947316 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.948560 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.948981 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.949068 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.949120 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.949365 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.949456 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:39:32.949666 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.945671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49135 (local address 127.0.98.2:46695)
0926 11:39:32.945818 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.945823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.946055 (+ 232us) server_negotiation.cc:407] Connection header received
0926 11:39:32.946124 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.946126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.946177 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.946260 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:32.946530 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.947316 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.948651 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.949078 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.949185 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.949247 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.949457 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.949532 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:39:32.949707 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.945457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46695 (local address 127.0.98.3:49135)
0926 11:39:32.945944 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:32.945957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:32.946051 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:32.946277 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:32.946280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:32.946291 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:32.946395 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.946403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.947488 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.947491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.948455 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:32.948466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.949225 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.949228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:32.949319 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.949349 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 11:39:32.949427 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 11:39:32.949488 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":126,"thread_start_us":144,"threads_started":1}
I20260926 11:39:32.950436 18137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb39b92b08a4eb7a5aab5f9ca5bbfa1" candidate_uuid: "df04096f11aa49abb090750421d9e5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" is_pre_election: true
I20260926 11:39:32.950594 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb39b92b08a4eb7a5aab5f9ca5bbfa1" candidate_uuid: "df04096f11aa49abb090750421d9e5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" is_pre_election: true
I20260926 11:39:32.950821 18137 raft_consensus.cc:2454] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df04096f11aa49abb090750421d9e5cf in term 0.
I20260926 11:39:32.950905 18268 raft_consensus.cc:2454] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df04096f11aa49abb090750421d9e5cf in term 0.
I20260926 11:39:32.951365 18334 leader_election.cc:302] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c222466b5f7a491b8eb69ab3b3afbcaf, df04096f11aa49abb090750421d9e5cf; no voters:
I20260926 11:39:32.951673 18469 raft_consensus.cc:2781] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:39:32.951758 18469 raft_consensus.cc:482] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:39:32.951812 18469 raft_consensus.cc:3037] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:32.952878 18469 raft_consensus.cc:504] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } }
I20260926 11:39:32.953286 18469 leader_election.cc:288] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [CANDIDATE]: Term 1 election: Requested vote from peers c222466b5f7a491b8eb69ab3b3afbcaf (127.0.98.2:46695), 58f1daee2cd14c2fbadeae687ad1c7f3 (127.0.98.1:39403)
I20260926 11:39:32.953687 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb39b92b08a4eb7a5aab5f9ca5bbfa1" candidate_uuid: "df04096f11aa49abb090750421d9e5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf"
I20260926 11:39:32.953718 18137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb39b92b08a4eb7a5aab5f9ca5bbfa1" candidate_uuid: "df04096f11aa49abb090750421d9e5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3"
I20260926 11:39:32.953840 18268 raft_consensus.cc:3037] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:32.953863 18137 raft_consensus.cc:3037] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:32.954901 18137 raft_consensus.cc:2454] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df04096f11aa49abb090750421d9e5cf in term 1.
I20260926 11:39:32.954902 18268 raft_consensus.cc:2454] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df04096f11aa49abb090750421d9e5cf in term 1.
I20260926 11:39:32.955229 18335 leader_election.cc:302] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f1daee2cd14c2fbadeae687ad1c7f3, df04096f11aa49abb090750421d9e5cf; no voters:
I20260926 11:39:32.955415 18469 raft_consensus.cc:2781] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:32.955684 18469 raft_consensus.cc:686] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 1 LEADER]: Becoming Leader. State: Replica: df04096f11aa49abb090750421d9e5cf, State: Running, Role: LEADER
I20260926 11:39:32.956039 18469 consensus_queue.cc:237] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } }
I20260926 11:39:32.959695 17995 catalog_manager.cc:5495] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf reported cstate change: term changed from 0 to 1, leader changed from <none> to df04096f11aa49abb090750421d9e5cf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "df04096f11aa49abb090750421d9e5cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df04096f11aa49abb090750421d9e5cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260926 11:39:32.980868 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:32.977174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.0.98.3:35309)
0926 11:39:32.977337 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:32.977342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:32.977363 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:39:32.977447 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:32.977449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:32.977503 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:32.977620 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:39:32.977879 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.978380 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.979248 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:32.979467 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:32.979576 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:32.979960 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:39:32.980397 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:39:32.980534 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:32.980598 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 11:39:32.980652 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:39:32.986821 18268 raft_consensus.cc:1261] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [term 1 FOLLOWER]: Refusing update from remote peer df04096f11aa49abb090750421d9e5cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:32.986821 18137 raft_consensus.cc:1261] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3 [term 1 FOLLOWER]: Refusing update from remote peer df04096f11aa49abb090750421d9e5cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:32.987677 18469 consensus_queue.cc:1035] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f1daee2cd14c2fbadeae687ad1c7f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:32.987891 18472 consensus_queue.cc:1035] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c222466b5f7a491b8eb69ab3b3afbcaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:32.996937 18475 mvcc.cc:205] Tried to move back new op lower bound from 7333571678146101248 to 7333571678030680064. Current Snapshot: MvccSnapshot[applied={T|T < 7333571678146101248}]
I20260926 11:39:32.997884 18476 mvcc.cc:205] Tried to move back new op lower bound from 7333571678146101248 to 7333571678030680064. Current Snapshot: MvccSnapshot[applied={T|T < 7333571678146101248}]
I20260926 11:39:33.006863 17995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:33.007043 17995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:33.009892 17995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48e63f626b544e0897e752d44f60d0cc: Sending DeleteTablet for 3 replicas of tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1
I20260926 11:39:33.010854 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:33 UTC) from {username='slave'} at 127.0.0.1:53980
I20260926 11:39:33.010911 18117 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:33 UTC) from {username='slave'} at 127.0.0.1:40502
I20260926 11:39:33.011116 18379 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:33 UTC) from {username='slave'} at 127.0.0.1:58060
I20260926 11:39:33.011656 18486 tablet_replica.cc:330] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3: stopping tablet replica
I20260926 11:39:33.011860 18487 tablet_replica.cc:330] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: stopping tablet replica
I20260926 11:39:33.012006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 18055
I20260926 11:39:33.012042 18486 raft_consensus.cc:2229] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P 58f1daee2cd14c2fbadeae687ad1c7f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:33.013581 18485 tablet_replica.cc:330] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: stopping tablet replica
I20260926 11:39:33.013959 18485 raft_consensus.cc:2229] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:33.014392 18485 raft_consensus.cc:2258] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:33.015476 18487 raft_consensus.cc:2229] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:39:33.016038 18487 raft_consensus.cc:2258] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:33.017748 18487 ts_tablet_manager.cc:1905] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:33.017748 18485 ts_tablet_manager.cc:1905] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 11:39:33.020586 17982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39403 (error 108)
I20260926 11:39:33.021066 18485 ts_tablet_manager.cc:1918] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:33.021184 18485 log.cc:1196] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/8bb39b92b08a4eb7a5aab5f9ca5bbfa1
I20260926 11:39:33.021122 18487 ts_tablet_manager.cc:1918] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:33.021437 18487 log.cc:1196] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/8bb39b92b08a4eb7a5aab5f9ca5bbfa1
I20260926 11:39:33.021467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 18186
I20260926 11:39:33.021508 18485 ts_tablet_manager.cc:1939] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P c222466b5f7a491b8eb69ab3b3afbcaf: Deleting consensus metadata
I20260926 11:39:33.021881 18487 ts_tablet_manager.cc:1939] T 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 P df04096f11aa49abb090750421d9e5cf: Deleting consensus metadata
I20260926 11:39:33.023909 17982 catalog_manager.cc:4841] TS df04096f11aa49abb090750421d9e5cf (127.0.98.3:35309): tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1 (table test-table [id=3e7a310c635243cc839ef400134e1a97]) successfully deleted
I20260926 11:39:33.024886 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:33.024504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39403 (local address 127.0.0.1:40516)
0926 11:39:33.024687 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:33.024777 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 11:39:33.025162 17982 catalog_manager.cc:4584] TS 58f1daee2cd14c2fbadeae687ad1c7f3 (127.0.98.1:39403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bb39b92b08a4eb7a5aab5f9ca5bbfa1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39403: connect: Connection refused (error 111)
I20260926 11:39:33.028820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 18317
I20260926 11:39:33.030532 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:33.029052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46695 (local address 127.0.0.1:53994)
0926 11:39:33.030345 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:33.030415 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1176}
I20260926 11:39:33.034226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 17964
2026-09-26T11:39:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1162 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:33 dist-test-slave-4pw5 krb5kdc[18491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:33 dist-test-slave-4pw5 krb5kdc[18491](info): set up 2 sockets
Sep 26 11:39:33 dist-test-slave-4pw5 krb5kdc[18491](info): commencing operation
krb5kdc: starting...
W20260926 11:39:34.115869 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:34 dist-test-slave-4pw5 krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:39:34.255779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37577
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:34.373719 18504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:34.374073 18504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:34.374148 18504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:34.378414 18504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:34.378520 18504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:34.378561 18504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:34.378612 18504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:34.378646 18504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:34.384428 18504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:34.385671 18504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:34.386878 18504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:34.396639 18512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:34.396653 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:34.396711 18509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:34.396975 18504 server_base.cc:1034] running on GCE node
I20260926 11:39:34.397791 18504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:34.399134 18504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:34.400398 18504 hybrid_clock.cc:648] HybridClock initialized: now 1790422774400385 us; error 40 us; skew 500 ppm
Sep 26 11:39:34 dist-test-slave-4pw5 krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:34.403855 18504 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:39:34.405298 18504 webserver.cc:460] Webserver started at http://127.0.98.62:41093/ using document root <none> and password file <none>
I20260926 11:39:34.405920 18504 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:34.405987 18504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:34.406183 18504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:34.408633 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "04c2050916e24a25923958411896eb15"
format_stamp: "Formatted at 2026-09-26 11:39:34 on dist-test-slave-4pw5"
server_key: "d6c8241bc484ddbcdb1a677559d8fd4c"
server_key_iv: "c0465f985acbe5b77e7cded2ce810456"
server_key_version: "encryptionkey@0"
I20260926 11:39:34.409307 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "04c2050916e24a25923958411896eb15"
format_stamp: "Formatted at 2026-09-26 11:39:34 on dist-test-slave-4pw5"
server_key: "d6c8241bc484ddbcdb1a677559d8fd4c"
server_key_iv: "c0465f985acbe5b77e7cded2ce810456"
server_key_version: "encryptionkey@0"
I20260926 11:39:34.413710 18504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 11:39:34.416676 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:34.418129 18504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:34.418287 18504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "04c2050916e24a25923958411896eb15"
format_stamp: "Formatted at 2026-09-26 11:39:34 on dist-test-slave-4pw5"
server_key: "d6c8241bc484ddbcdb1a677559d8fd4c"
server_key_iv: "c0465f985acbe5b77e7cded2ce810456"
server_key_version: "encryptionkey@0"
I20260926 11:39:34.418401 18504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:34.430778 18504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:34.440521 18504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:34.440711 18504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:34.448668 18504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37577
I20260926 11:39:34.448705 18571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37577 every 8 connection(s)
I20260926 11:39:34.449791 18504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:34.453238 18572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:34.453508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 18504
I20260926 11:39:34.453663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:34.453996 392 external_mini_cluster.cc:1428] Setting key fce20e31eeaef796f1304d5f73f2d766
I20260926 11:39:34.460641 18572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15: Bootstrap starting.
I20260926 11:39:34.463219 18572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15: Neither blocks nor log segments found. Creating new log.
Sep 26 11:39:34 dist-test-slave-4pw5 krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:39:34.464130 18572 log.cc:824] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:34.466976 18572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15: No bootstrap required, opened a new log
I20260926 11:39:34.470419 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:34.455532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.0.98.62:37577)
0926 11:39:34.456153 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:34.456171 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:34.456210 (+ 39us) server_negotiation.cc:407] Connection header received
0926 11:39:34.456950 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:34.456972 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:34.457351 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:34.457732 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:34.458618 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:34.459743 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:34.460638 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:34.461200 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:34.461380 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:34.464205 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:34.464212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:34.464233 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:34.464273 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:34.466679 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:34.467384 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:34.467393 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:34.467400 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:34.467544 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:34.467897 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:34.467900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:34.467902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:34.468466 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:34.468652 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:34.468981 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 11:39:34.469351 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":179,"threads_started":1}
I20260926 11:39:34.471992 18572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c2050916e24a25923958411896eb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37577 } }
I20260926 11:39:34.472242 18572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:34.472317 18572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c2050916e24a25923958411896eb15, State: Initialized, Role: FOLLOWER
I20260926 11:39:34.472827 18572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c2050916e24a25923958411896eb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37577 } }
I20260926 11:39:34.473011 18572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:34.473111 18572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:34.473217 18572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:34.474267 18572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c2050916e24a25923958411896eb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37577 } }
I20260926 11:39:34.474632 18572 leader_election.cc:302] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04c2050916e24a25923958411896eb15; no voters:
I20260926 11:39:34.474980 18572 leader_election.cc:288] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:34.475169 18577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:34.475471 18577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [term 1 LEADER]: Becoming Leader. State: Replica: 04c2050916e24a25923958411896eb15, State: Running, Role: LEADER
I20260926 11:39:34.475847 18577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c2050916e24a25923958411896eb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37577 } }
I20260926 11:39:34.476359 18572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:34.477705 18578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04c2050916e24a25923958411896eb15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c2050916e24a25923958411896eb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37577 } } }
I20260926 11:39:34.477844 18578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:34.478186 18579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c2050916e24a25923958411896eb15. Latest consensus state: current_term: 1 leader_uuid: "04c2050916e24a25923958411896eb15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c2050916e24a25923958411896eb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37577 } } }
I20260926 11:39:34.478293 18579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:34.478411 18586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:34.481514 18586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:34.487390 18586 catalog_manager.cc:1347] Generated new cluster ID: 2fa67d8cfcd04903be5b76b12bae69b1
I20260926 11:39:34.487468 18586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:34.518862 18586 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:34.519975 18586 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:34.527360 18586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15: Generated new TSK 0
I20260926 11:39:34.528188 18586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:39:34.563197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37577
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:34.701988 18598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:34.702282 18598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:34.702328 18598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:34.706422 18598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:34.706493 18598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:34.706553 18598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:39:34.711979 18598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:34.713047 18598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:34.714025 18598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:34.724159 18604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:34.724159 18603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:34.724222 18598 server_base.cc:1034] running on GCE node
W20260926 11:39:34.724159 18606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:34.724905 18598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:34.725739 18598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:34.726950 18598 hybrid_clock.cc:648] HybridClock initialized: now 1790422774726918 us; error 46 us; skew 500 ppm
Sep 26 11:39:34 dist-test-slave-4pw5 krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:34.730104 18598 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:39:34.731521 18598 webserver.cc:460] Webserver started at http://127.0.98.1:38211/ using document root <none> and password file <none>
I20260926 11:39:34.732224 18598 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:34.732295 18598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:34.732534 18598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:34.734766 18598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "77e6842f9b704274bdd5aabb8a0f6f65"
format_stamp: "Formatted at 2026-09-26 11:39:34 on dist-test-slave-4pw5"
server_key: "00d3874a4744b92c4dc86a2ddd3861ea"
server_key_iv: "ff6173eb63d5e0e0bf343cb6dbfbb924"
server_key_version: "encryptionkey@0"
I20260926 11:39:34.735368 18598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "77e6842f9b704274bdd5aabb8a0f6f65"
format_stamp: "Formatted at 2026-09-26 11:39:34 on dist-test-slave-4pw5"
server_key: "00d3874a4744b92c4dc86a2ddd3861ea"
server_key_iv: "ff6173eb63d5e0e0bf343cb6dbfbb924"
server_key_version: "encryptionkey@0"
I20260926 11:39:34.740350 18598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 11:39:34.743664 18613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:34.745158 18598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:39:34.745366 18598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "77e6842f9b704274bdd5aabb8a0f6f65"
format_stamp: "Formatted at 2026-09-26 11:39:34 on dist-test-slave-4pw5"
server_key: "00d3874a4744b92c4dc86a2ddd3861ea"
server_key_iv: "ff6173eb63d5e0e0bf343cb6dbfbb924"
server_key_version: "encryptionkey@0"
I20260926 11:39:34.745515 18598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:34.759442 18598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:34.763127 18598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:34.763366 18598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:34.764081 18598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:34.765151 18598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:34.765276 18598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:34.765357 18598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:34.765403 18598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:34.775384 18598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36603
I20260926 11:39:34.775404 18726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36603 every 8 connection(s)
I20260926 11:39:34.776408 18598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:39:34.781494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 18598
I20260926 11:39:34.781605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:39:34.781916 392 external_mini_cluster.cc:1428] Setting key 2af9ad606d6e930667e24007f7124bc0
Sep 26 11:39:34 dist-test-slave-4pw5 krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:39:34.793876 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:34.778457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53391 (local address 127.0.98.62:37577)
0926 11:39:34.778672 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:34.778676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:34.779487 (+ 811us) server_negotiation.cc:407] Connection header received
0926 11:39:34.780939 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:34.780942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:34.780989 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:34.781064 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:34.782613 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:34.783399 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:34.785548 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:34.785988 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:34.786118 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:34.789301 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:34.789307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:34.789313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:34.789352 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:34.791260 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:34.791981 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:34.791985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:34.791987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:34.792084 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:34.792644 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:34.792647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:34.792648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:34.792863 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:34.792947 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:34.793584 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 11:39:34.793739 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 11:39:34.794723 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:34.778855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37577 (local address 127.0.98.1:53391)
0926 11:39:34.779320 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:34.779368 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:34.780712 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:34.781277 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:34.781284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:34.781827 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:34.782438 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:34.782460 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:34.783672 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:34.783677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:34.785390 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:34.785406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:34.786192 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:34.786196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:34.786363 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:34.787156 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:34.787177 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:34.789079 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:34.791412 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:34.791418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:34.791434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:34.791839 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:34.792251 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:34.792254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:34.792255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:34.792495 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:34.792986 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:34.792992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:34.793330 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 11:39:34.793504 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 11:39:34.793797 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:39:34.796272 18727 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37577
I20260926 11:39:34.796569 18727 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:34.797148 18727 heartbeater.cc:511] Master 127.0.98.62:37577 requested a full tablet report, sending...
I20260926 11:39:34.799014 18536 ts_manager.cc:194] Registered new tserver with Master: 77e6842f9b704274bdd5aabb8a0f6f65 (127.0.98.1:36603)
I20260926 11:39:34.800948 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:39:34.816735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37577
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:34.934360 18732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:34.934669 18732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:34.934715 18732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:34.939042 18732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:34.939131 18732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:34.939195 18732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:39:34.944855 18732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:34.946092 18732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:34.947183 18732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:34.957854 18737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:34.958029 18732 server_base.cc:1034] running on GCE node
W20260926 11:39:34.957894 18740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:34.957926 18738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:34.958626 18732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:34.959295 18732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:34.960469 18732 hybrid_clock.cc:648] HybridClock initialized: now 1790422774960446 us; error 36 us; skew 500 ppm
Sep 26 11:39:34 dist-test-slave-4pw5 krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:34.963549 18732 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:39:34.964591 18732 webserver.cc:460] Webserver started at http://127.0.98.2:35719/ using document root <none> and password file <none>
I20260926 11:39:34.965197 18732 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:34.965296 18732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:34.965533 18732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:34.967796 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "24233f9f24cc4f24ad18ffebb733c5d6"
format_stamp: "Formatted at 2026-09-26 11:39:34 on dist-test-slave-4pw5"
server_key: "e8ec64a07aa601d1c8a0af6318544e8d"
server_key_iv: "5b036ed72644888155e50d250d7fed7e"
server_key_version: "encryptionkey@0"
I20260926 11:39:34.968385 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "24233f9f24cc4f24ad18ffebb733c5d6"
format_stamp: "Formatted at 2026-09-26 11:39:34 on dist-test-slave-4pw5"
server_key: "e8ec64a07aa601d1c8a0af6318544e8d"
server_key_iv: "5b036ed72644888155e50d250d7fed7e"
server_key_version: "encryptionkey@0"
I20260926 11:39:34.972550 18732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:34.975189 18747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:34.976576 18732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:34.976727 18732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "24233f9f24cc4f24ad18ffebb733c5d6"
format_stamp: "Formatted at 2026-09-26 11:39:34 on dist-test-slave-4pw5"
server_key: "e8ec64a07aa601d1c8a0af6318544e8d"
server_key_iv: "5b036ed72644888155e50d250d7fed7e"
server_key_version: "encryptionkey@0"
I20260926 11:39:34.976899 18732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:34.991122 18732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:34.994786 18732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:34.995002 18732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:34.995684 18732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:34.996596 18732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:34.996671 18732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:34.996754 18732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:34.996797 18732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:35.007164 18732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38645
I20260926 11:39:35.007186 18860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38645 every 8 connection(s)
I20260926 11:39:35.008301 18732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:39:35.014019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 18732
I20260926 11:39:35.014154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:39:35.014446 392 external_mini_cluster.cc:1428] Setting key c2c64e8a508c2bfbe28a8549327e64a7
Sep 26 11:39:35 dist-test-slave-4pw5 krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:39:35.025034 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.010270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56793 (local address 127.0.98.62:37577)
0926 11:39:35.010423 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.010427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.011326 (+ 899us) server_negotiation.cc:407] Connection header received
0926 11:39:35.012900 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.012904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.012968 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.013075 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:35.014684 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.015366 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.017052 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.017467 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.017606 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.020795 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:35.020798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:35.020801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:35.020830 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:35.022579 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:35.023245 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:35.023251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:35.023257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:35.023341 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:35.023781 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:35.023785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:35.023786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:35.023992 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:35.024105 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.024741 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.024902 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:39:35.026012 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.010693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37577 (local address 127.0.98.2:56793)
0926 11:39:35.011159 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:35.011209 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:35.012647 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:35.013267 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:35.013276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:35.013680 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:35.014437 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.014459 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.015501 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.015504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.016886 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.016899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.017752 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.017758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.018019 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.018758 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:35.018781 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:35.020568 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:35.022738 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:35.022746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:35.022766 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:35.023126 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:35.023449 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:35.023452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:35.023454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:35.023681 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:35.024104 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:35.024113 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:35.024414 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 11:39:35.024617 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 11:39:35.024938 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:39:35.027468 18861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37577
I20260926 11:39:35.027786 18861 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:35.028469 18861 heartbeater.cc:511] Master 127.0.98.62:37577 requested a full tablet report, sending...
I20260926 11:39:35.029800 18536 ts_manager.cc:194] Registered new tserver with Master: 24233f9f24cc4f24ad18ffebb733c5d6 (127.0.98.2:38645)
I20260926 11:39:35.030413 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:39:35.043310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37577
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:35.159153 18866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:35.159487 18866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:35.159601 18866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:35.164189 18866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:35.164315 18866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:35.164417 18866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:39:35.170329 18866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:35.171742 18866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:35.172993 18866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:35.183646 18874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:35.183646 18872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:35.183650 18871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:35.184053 18866 server_base.cc:1034] running on GCE node
I20260926 11:39:35.184525 18866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:35.185618 18866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:35.186863 18866 hybrid_clock.cc:648] HybridClock initialized: now 1790422775186853 us; error 32 us; skew 500 ppm
Sep 26 11:39:35 dist-test-slave-4pw5 krb5kdc[18491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:35.190236 18866 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:39:35.191303 18866 webserver.cc:460] Webserver started at http://127.0.98.3:44561/ using document root <none> and password file <none>
I20260926 11:39:35.192030 18866 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:35.192137 18866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:35.192497 18866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:35.195302 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ecf7330ad6a4b49bd5c3be67c660275"
format_stamp: "Formatted at 2026-09-26 11:39:35 on dist-test-slave-4pw5"
server_key: "66c3c069f13032f7f3d4b215f4c26621"
server_key_iv: "c07a249b30f9294b44d9d907aac73dcc"
server_key_version: "encryptionkey@0"
I20260926 11:39:35.196122 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ecf7330ad6a4b49bd5c3be67c660275"
format_stamp: "Formatted at 2026-09-26 11:39:35 on dist-test-slave-4pw5"
server_key: "66c3c069f13032f7f3d4b215f4c26621"
server_key_iv: "c07a249b30f9294b44d9d907aac73dcc"
server_key_version: "encryptionkey@0"
I20260926 11:39:35.201825 18866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:39:35.204629 18881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:35.206033 18866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:35.206197 18866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "5ecf7330ad6a4b49bd5c3be67c660275"
format_stamp: "Formatted at 2026-09-26 11:39:35 on dist-test-slave-4pw5"
server_key: "66c3c069f13032f7f3d4b215f4c26621"
server_key_iv: "c07a249b30f9294b44d9d907aac73dcc"
server_key_version: "encryptionkey@0"
I20260926 11:39:35.206346 18866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:35.223196 18866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:35.226707 18866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:35.226918 18866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:35.227521 18866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:35.228406 18866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:35.228452 18866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:35.228493 18866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:35.228509 18866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:35.238129 18866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43257
I20260926 11:39:35.238247 18994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43257 every 8 connection(s)
I20260926 11:39:35.239122 18866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:39:35.240723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 18866
I20260926 11:39:35.240868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:39:35.241241 392 external_mini_cluster.cc:1428] Setting key 4ce9ea43db1a18ddd9fe983fdee84c0b
Sep 26 11:39:35 dist-test-slave-4pw5 krb5kdc[18491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:39:35.254267 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.241275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51409 (local address 127.0.98.62:37577)
0926 11:39:35.241422 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.241427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.242380 (+ 953us) server_negotiation.cc:407] Connection header received
0926 11:39:35.243876 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.243879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.243924 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.243992 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:35.245123 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.245678 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.246858 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.247073 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.247152 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.250136 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:35.250139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:35.250141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:35.250190 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:35.251850 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:35.252437 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:35.252440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:35.252442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:35.252555 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:35.252950 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:35.252953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:35.252955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:35.253184 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:35.253283 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.253966 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.254080 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:39:35.255362 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.241715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37577 (local address 127.0.98.3:51409)
0926 11:39:35.242240 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:35.242292 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:35.243689 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:35.244124 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:35.244130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:35.244485 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:35.244981 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.244994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.245821 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.245824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.246742 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.246752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.247212 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.247215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.247357 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.248047 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:35.248064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:35.249952 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:35.251974 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:35.251980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:35.251996 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:35.252295 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:35.252668 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:35.252670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:35.252671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:35.252834 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:35.253288 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:35.253294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:35.253647 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 11:39:35.253867 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 11:39:35.254245 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":152,"threads_started":1}
I20260926 11:39:35.256551 18995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37577
I20260926 11:39:35.256871 18995 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:35.257471 18995 heartbeater.cc:511] Master 127.0.98.62:37577 requested a full tablet report, sending...
I20260926 11:39:35.258627 18536 ts_manager.cc:194] Registered new tserver with Master: 5ecf7330ad6a4b49bd5c3be67c660275 (127.0.98.3:43257)
I20260926 11:39:35.259263 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51409
I20260926 11:39:35.268062 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:39:35.279392 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.270346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.0.98.62:37577)
0926 11:39:35.270488 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.270492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.270657 (+ 165us) server_negotiation.cc:407] Connection header received
0926 11:39:35.270796 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.270798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.270840 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.270914 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:35.271718 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.272385 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.273336 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.273560 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.273647 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.275310 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:35.275313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:35.275316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:35.275344 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:35.276993 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:35.277804 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:35.277807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:35.277809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:35.277888 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:35.278336 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:35.278339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:35.278341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:35.278680 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:35.278809 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.279174 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.279264 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:39:35.283236 18536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:39:35.286589 18536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:39:35.303503 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.297519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.0.98.2:38645)
0926 11:39:35.298108 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.298117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.298135 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:39:35.298234 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.298238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.298422 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.298857 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:35.299018 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.300127 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.301598 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.302492 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.302619 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.302783 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.303217 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.303327 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":100,"threads_started":1}
I20260926 11:39:35.303571 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.296507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38645 (local address 127.0.0.1:51316)
0926 11:39:35.297028 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:35.297100 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:35.297198 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:35.298570 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:35.298573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:35.298590 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:35.298697 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.298703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.300270 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.300273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.301465 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.301475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.302669 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.302673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.302943 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.303055 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 11:39:35.303164 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 11:39:35.303402 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":195,"thread_start_us":135,"threads_started":1}
I20260926 11:39:35.303926 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.296297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36603 (local address 127.0.0.1:50072)
0926 11:39:35.296814 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:35.296871 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:35.297008 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:35.298245 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:35.298251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:35.298276 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:35.298415 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.298422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.300004 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.300008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.301633 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.301646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.302669 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.302673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.302946 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.303023 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 11:39:35.303164 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 11:39:35.303777 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":296,"thread_start_us":69,"threads_started":1}
I20260926 11:39:35.304420 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.297147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43257 (local address 127.0.0.1:51206)
0926 11:39:35.297815 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:35.297833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:35.297923 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:35.298432 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:35.298435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:35.298457 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:35.298559 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.298570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.299751 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.299757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.301633 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.301646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.303972 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.303975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.304070 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.304098 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 11:39:35.304176 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 11:39:35.304296 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":260,"thread_start_us":109,"threads_started":1}
I20260926 11:39:35.304916 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.297524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.0.98.3:43257)
0926 11:39:35.297964 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.297975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.297998 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:39:35.298075 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.298080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.298260 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.298406 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:35.298703 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.299527 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.302863 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.303643 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.303775 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.303908 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.304654 (+ 746us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.304764 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":105,"threads_started":1}
I20260926 11:39:35.305423 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.296729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.0.98.1:36603)
0926 11:39:35.297677 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.297687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.297714 (+ 27us) server_negotiation.cc:407] Connection header received
0926 11:39:35.297801 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.297807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.298065 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.298233 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:35.298571 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.299870 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.301774 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.302510 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.302619 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.302783 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.305161 (+ 2378us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.305283 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":63,"threads_started":1}
I20260926 11:39:35.307658 18661 tablet_service.cc:1467] Processing CreateTablet for tablet a2da72e0da9941b0b82a0f53cdcfb511 (DEFAULT_TABLE table=test-table [id=186fd98adffa42e3ae1f15f0e71fb495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:35.307651 18929 tablet_service.cc:1467] Processing CreateTablet for tablet a2da72e0da9941b0b82a0f53cdcfb511 (DEFAULT_TABLE table=test-table [id=186fd98adffa42e3ae1f15f0e71fb495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:35.307657 18795 tablet_service.cc:1467] Processing CreateTablet for tablet a2da72e0da9941b0b82a0f53cdcfb511 (DEFAULT_TABLE table=test-table [id=186fd98adffa42e3ae1f15f0e71fb495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:35.308848 18929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2da72e0da9941b0b82a0f53cdcfb511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:35.308848 18795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2da72e0da9941b0b82a0f53cdcfb511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:35.308849 18661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2da72e0da9941b0b82a0f53cdcfb511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:35.315805 19012 tablet_bootstrap.cc:492] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65: Bootstrap starting.
I20260926 11:39:35.317678 19014 tablet_bootstrap.cc:492] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: Bootstrap starting.
I20260926 11:39:35.317679 19013 tablet_bootstrap.cc:492] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: Bootstrap starting.
I20260926 11:39:35.318302 19012 tablet_bootstrap.cc:654] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:35.319233 19012 log.cc:824] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:35.320031 19013 tablet_bootstrap.cc:654] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:35.320026 19014 tablet_bootstrap.cc:654] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:35.321431 19013 log.cc:824] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:35.321448 19014 log.cc:824] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:35.321494 19012 tablet_bootstrap.cc:492] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65: No bootstrap required, opened a new log
I20260926 11:39:35.321743 19012 ts_tablet_manager.cc:1397] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:39:35.323674 19013 tablet_bootstrap.cc:492] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: No bootstrap required, opened a new log
I20260926 11:39:35.323801 19014 tablet_bootstrap.cc:492] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: No bootstrap required, opened a new log
I20260926 11:39:35.323872 19013 ts_tablet_manager.cc:1397] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:39:35.323952 19014 ts_tablet_manager.cc:1397] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:39:35.325801 19012 raft_consensus.cc:348] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } }
I20260926 11:39:35.326350 19012 raft_consensus.cc:374] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:35.326483 19012 raft_consensus.cc:729] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e6842f9b704274bdd5aabb8a0f6f65, State: Initialized, Role: FOLLOWER
I20260926 11:39:35.327064 19012 consensus_queue.cc:260] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } }
I20260926 11:39:35.327967 18727 heartbeater.cc:503] Master 127.0.98.62:37577 was elected leader, sending a full tablet report...
I20260926 11:39:35.327924 19013 raft_consensus.cc:348] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } }
I20260926 11:39:35.327924 19014 raft_consensus.cc:348] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } }
I20260926 11:39:35.328215 19014 raft_consensus.cc:374] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:35.328302 19013 raft_consensus.cc:374] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:35.328313 19014 raft_consensus.cc:729] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ecf7330ad6a4b49bd5c3be67c660275, State: Initialized, Role: FOLLOWER
I20260926 11:39:35.328409 19013 raft_consensus.cc:729] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24233f9f24cc4f24ad18ffebb733c5d6, State: Initialized, Role: FOLLOWER
I20260926 11:39:35.328439 19012 ts_tablet_manager.cc:1428] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260926 11:39:35.328814 19014 consensus_queue.cc:260] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } }
I20260926 11:39:35.328823 19013 consensus_queue.cc:260] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } }
I20260926 11:39:35.329522 18861 heartbeater.cc:503] Master 127.0.98.62:37577 was elected leader, sending a full tablet report...
I20260926 11:39:35.329840 19014 ts_tablet_manager.cc:1428] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:39:35.330502 18995 heartbeater.cc:503] Master 127.0.98.62:37577 was elected leader, sending a full tablet report...
I20260926 11:39:35.332231 19013 ts_tablet_manager.cc:1428] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260926 11:39:35.394116 19020 raft_consensus.cc:482] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:35.394371 19020 raft_consensus.cc:504] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } }
I20260926 11:39:35.395869 19020 leader_election.cc:288] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e6842f9b704274bdd5aabb8a0f6f65 (127.0.98.1:36603), 24233f9f24cc4f24ad18ffebb733c5d6 (127.0.98.2:38645)
I20260926 11:39:35.401475 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.396469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42837 (local address 127.0.98.2:38645)
0926 11:39:35.396662 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.396667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.396713 (+ 46us) server_negotiation.cc:407] Connection header received
0926 11:39:35.396779 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.396782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.396830 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.396929 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:35.397294 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.398390 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.400282 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.400751 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.400853 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.400916 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.401223 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.401317 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:39:35.401479 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.396114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38645 (local address 127.0.98.3:42837)
0926 11:39:35.396322 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:35.396338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:35.396433 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:35.396967 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:35.396970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:35.396992 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:35.397141 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.397150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.398536 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.398539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.400091 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.400105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.400921 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.400924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.401034 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.401058 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:39:35.401197 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 11:39:35.401296 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":41,"spinlock_wait_cycles":4352}
I20260926 11:39:35.402133 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.396339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57591 (local address 127.0.98.1:36603)
0926 11:39:35.396459 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.396467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.396968 (+ 501us) server_negotiation.cc:407] Connection header received
0926 11:39:35.397225 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.397229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.397284 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.397376 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:35.397714 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.398554 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.400980 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.401479 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.401618 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.401687 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.401926 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.402008 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:39:35.402153 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.396114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36603 (local address 127.0.98.3:57591)
0926 11:39:35.396858 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:35.396869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:35.397049 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:35.397407 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:35.397410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:35.397430 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:35.397540 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.397547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.398721 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.398724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.400091 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.400105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.401653 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.401658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.401786 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.401808 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:39:35.401908 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 11:39:35.401985 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":424,"spinlock_wait_cycles":2688,"thread_start_us":170,"threads_started":1}
I20260926 11:39:35.402539 18813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2da72e0da9941b0b82a0f53cdcfb511" candidate_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" is_pre_election: true
I20260926 11:39:35.402745 18681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2da72e0da9941b0b82a0f53cdcfb511" candidate_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" is_pre_election: true
I20260926 11:39:35.402947 18813 raft_consensus.cc:2454] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ecf7330ad6a4b49bd5c3be67c660275 in term 0.
I20260926 11:39:35.403071 18681 raft_consensus.cc:2454] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ecf7330ad6a4b49bd5c3be67c660275 in term 0.
I20260926 11:39:35.403646 18883 leader_election.cc:302] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ecf7330ad6a4b49bd5c3be67c660275, 77e6842f9b704274bdd5aabb8a0f6f65; no voters:
I20260926 11:39:35.403997 19020 raft_consensus.cc:2781] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:39:35.404100 19020 raft_consensus.cc:482] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:39:35.404136 19020 raft_consensus.cc:3037] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:35.405483 19020 raft_consensus.cc:504] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } }
I20260926 11:39:35.405946 19020 leader_election.cc:288] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [CANDIDATE]: Term 1 election: Requested vote from peers 77e6842f9b704274bdd5aabb8a0f6f65 (127.0.98.1:36603), 24233f9f24cc4f24ad18ffebb733c5d6 (127.0.98.2:38645)
I20260926 11:39:35.406406 18813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2da72e0da9941b0b82a0f53cdcfb511" candidate_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24233f9f24cc4f24ad18ffebb733c5d6"
I20260926 11:39:35.406462 18681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2da72e0da9941b0b82a0f53cdcfb511" candidate_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e6842f9b704274bdd5aabb8a0f6f65"
I20260926 11:39:35.406571 18813 raft_consensus.cc:3037] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:35.406622 18681 raft_consensus.cc:3037] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:35.407814 18813 raft_consensus.cc:2454] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ecf7330ad6a4b49bd5c3be67c660275 in term 1.
I20260926 11:39:35.407819 18681 raft_consensus.cc:2454] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ecf7330ad6a4b49bd5c3be67c660275 in term 1.
I20260926 11:39:35.408221 18882 leader_election.cc:302] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24233f9f24cc4f24ad18ffebb733c5d6, 5ecf7330ad6a4b49bd5c3be67c660275; no voters:
I20260926 11:39:35.408422 19020 raft_consensus.cc:2781] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:35.408712 19020 raft_consensus.cc:686] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 1 LEADER]: Becoming Leader. State: Replica: 5ecf7330ad6a4b49bd5c3be67c660275, State: Running, Role: LEADER
I20260926 11:39:35.409103 19020 consensus_queue.cc:237] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } }
I20260926 11:39:35.412778 18535 catalog_manager.cc:5495] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ecf7330ad6a4b49bd5c3be67c660275 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ecf7330ad6a4b49bd5c3be67c660275" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43257 } health_report { overall_health: HEALTHY } } }
I20260926 11:39:35.429745 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.425836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.0.98.3:43257)
0926 11:39:35.425998 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.426003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.426072 (+ 69us) server_negotiation.cc:407] Connection header received
0926 11:39:35.426152 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.426154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.426212 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.426309 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:39:35.426579 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.427087 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.428106 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.428349 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.428435 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.428745 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:39:35.429180 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:39:35.429339 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.429494 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.429565 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 11:39:35.437175 18813 raft_consensus.cc:1261] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [term 1 FOLLOWER]: Refusing update from remote peer 5ecf7330ad6a4b49bd5c3be67c660275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:35.437175 18681 raft_consensus.cc:1261] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65 [term 1 FOLLOWER]: Refusing update from remote peer 5ecf7330ad6a4b49bd5c3be67c660275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:35.438099 19020 consensus_queue.cc:1035] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e6842f9b704274bdd5aabb8a0f6f65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:35.438387 19022 consensus_queue.cc:1035] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24233f9f24cc4f24ad18ffebb733c5d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:35.447311 19025 mvcc.cc:205] Tried to move back new op lower bound from 7333571688179433472 to 7333571688078761984. Current Snapshot: MvccSnapshot[applied={T|T < 7333571688179433472}]
I20260926 11:39:35.447880 19026 mvcc.cc:205] Tried to move back new op lower bound from 7333571688179433472 to 7333571688078761984. Current Snapshot: MvccSnapshot[applied={T|T < 7333571688179433472}]
I20260926 11:39:35.460582 18534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:35.460795 18534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:35.463850 18534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04c2050916e24a25923958411896eb15: Sending DeleteTablet for 3 replicas of tablet a2da72e0da9941b0b82a0f53cdcfb511
I20260926 11:39:35.464891 18661 tablet_service.cc:1514] Processing DeleteTablet for tablet a2da72e0da9941b0b82a0f53cdcfb511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50072
I20260926 11:39:35.464995 18929 tablet_service.cc:1514] Processing DeleteTablet for tablet a2da72e0da9941b0b82a0f53cdcfb511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51206
I20260926 11:39:35.465489 18795 tablet_service.cc:1514] Processing DeleteTablet for tablet a2da72e0da9941b0b82a0f53cdcfb511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51316
I20260926 11:39:35.465562 19035 tablet_replica.cc:330] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65: stopping tablet replica
I20260926 11:39:35.465977 19035 raft_consensus.cc:2229] T a2da72e0da9941b0b82a0f53cdcfb511 P 77e6842f9b704274bdd5aabb8a0f6f65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:35.466018 19037 tablet_replica.cc:330] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: stopping tablet replica
I20260926 11:39:35.466302 19036 tablet_replica.cc:330] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: stopping tablet replica
I20260926 11:39:35.466326 19037 raft_consensus.cc:2229] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:35.466676 19036 raft_consensus.cc:2229] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:39:35.466775 19037 raft_consensus.cc:2258] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:35.466830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 18598
I20260926 11:39:35.467612 19036 raft_consensus.cc:2258] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:35.469331 19036 ts_tablet_manager.cc:1905] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:35.469332 19037 ts_tablet_manager.cc:1905] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:35.472757 19036 ts_tablet_manager.cc:1918] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:35.472874 19036 log.cc:1196] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/a2da72e0da9941b0b82a0f53cdcfb511
I20260926 11:39:35.473246 19036 ts_tablet_manager.cc:1939] T a2da72e0da9941b0b82a0f53cdcfb511 P 5ecf7330ad6a4b49bd5c3be67c660275: Deleting consensus metadata
I20260926 11:39:35.473696 19037 ts_tablet_manager.cc:1918] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:35.473809 19037 log.cc:1196] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/a2da72e0da9941b0b82a0f53cdcfb511
I20260926 11:39:35.474154 19037 ts_tablet_manager.cc:1939] T a2da72e0da9941b0b82a0f53cdcfb511 P 24233f9f24cc4f24ad18ffebb733c5d6: Deleting consensus metadata
I20260926 11:39:35.474684 18522 catalog_manager.cc:4841] TS 5ecf7330ad6a4b49bd5c3be67c660275 (127.0.98.3:43257): tablet a2da72e0da9941b0b82a0f53cdcfb511 (table test-table [id=186fd98adffa42e3ae1f15f0e71fb495]) successfully deleted
I20260926 11:39:35.475750 18520 catalog_manager.cc:4841] TS 24233f9f24cc4f24ad18ffebb733c5d6 (127.0.98.2:38645): tablet a2da72e0da9941b0b82a0f53cdcfb511 (table test-table [id=186fd98adffa42e3ae1f15f0e71fb495]) successfully deleted
W20260926 11:39:35.479383 18521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36603 (error 108)
I20260926 11:39:35.479441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 18732
I20260926 11:39:35.480974 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.480361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36603 (local address 127.0.0.1:50080)
0926 11:39:35.480782 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:35.480862 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317}
W20260926 11:39:35.481263 18521 catalog_manager.cc:4584] TS 77e6842f9b704274bdd5aabb8a0f6f65 (127.0.98.1:36603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2da72e0da9941b0b82a0f53cdcfb511: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36603: connect: Connection refused (error 111)
I20260926 11:39:35.485984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 18866
I20260926 11:39:35.491420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 18504
2026-09-26T11:39:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2456 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T11:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:35Z Disabled control of system clock
I20260926 11:39:35.523093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40393
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:39:35.645843 19042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:35.646186 19042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:35.646234 19042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:35.650669 19042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:35.650755 19042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:35.650771 19042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:35.650789 19042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:35.650805 19042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:35.655987 19042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:35.657155 19042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:35.658556 19042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:35.667896 19048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:35.667896 19050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:35.667949 19047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:35.667997 19042 server_base.cc:1034] running on GCE node
I20260926 11:39:35.668872 19042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:35.670127 19042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:35.671331 19042 hybrid_clock.cc:648] HybridClock initialized: now 1790422775671301 us; error 54 us; skew 500 ppm
I20260926 11:39:35.673866 19042 webserver.cc:460] Webserver started at http://127.0.98.62:39783/ using document root <none> and password file <none>
I20260926 11:39:35.674495 19042 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:35.674569 19042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:35.674827 19042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:35.677096 19042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f"
format_stamp: "Formatted at 2026-09-26 11:39:35 on dist-test-slave-4pw5"
server_key: "e71e63bb3d0784555ef7ff2240f9b775"
server_key_iv: "73fb3f25acf93190d0e4488d306a833f"
server_key_version: "encryptionkey@0"
I20260926 11:39:35.677726 19042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f"
format_stamp: "Formatted at 2026-09-26 11:39:35 on dist-test-slave-4pw5"
server_key: "e71e63bb3d0784555ef7ff2240f9b775"
server_key_iv: "73fb3f25acf93190d0e4488d306a833f"
server_key_version: "encryptionkey@0"
I20260926 11:39:35.681838 19042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:39:35.684626 19056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:35.686029 19042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:35.686211 19042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f"
format_stamp: "Formatted at 2026-09-26 11:39:35 on dist-test-slave-4pw5"
server_key: "e71e63bb3d0784555ef7ff2240f9b775"
server_key_iv: "73fb3f25acf93190d0e4488d306a833f"
server_key_version: "encryptionkey@0"
I20260926 11:39:35.686360 19042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:35.711277 19042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:35.712157 19042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:35.712381 19042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:35.720916 19042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40393
I20260926 11:39:35.720928 19108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40393 every 8 connection(s)
I20260926 11:39:35.722034 19042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:35.725765 19109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:35.730674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 19042
I20260926 11:39:35.730808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:35.731204 392 external_mini_cluster.cc:1428] Setting key cd344991172dae7f74ddd5086ad39d5f
I20260926 11:39:35.732403 19109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f: Bootstrap starting.
I20260926 11:39:35.734835 19109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:35.735860 19109 log.cc:824] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:35.738511 19109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f: No bootstrap required, opened a new log
I20260926 11:39:35.742066 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.732947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.0.98.62:40393)
0926 11:39:35.733611 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.733626 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.733662 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:39:35.734345 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.734363 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.734775 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.735135 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:35.735509 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.736636 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.737676 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.738377 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.738554 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:35.739535 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:35.739542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:35.739559 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:35.740104 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:35.740181 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:35.740209 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:35.740326 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:35.740502 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 11:39:35.740806 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":151,"threads_started":1}
I20260926 11:39:35.742940 19109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } }
I20260926 11:39:35.743194 19109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:35.743230 19109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a412a2b0c0d49cab1b5d10dd91e2f5f, State: Initialized, Role: FOLLOWER
I20260926 11:39:35.743772 19109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } }
I20260926 11:39:35.743911 19109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:35.743999 19109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:35.744131 19109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:35.745198 19109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } }
I20260926 11:39:35.745589 19109 leader_election.cc:302] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a412a2b0c0d49cab1b5d10dd91e2f5f; no voters:
I20260926 11:39:35.745976 19109 leader_election.cc:288] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:35.746138 19114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:35.746466 19114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [term 1 LEADER]: Becoming Leader. State: Replica: 3a412a2b0c0d49cab1b5d10dd91e2f5f, State: Running, Role: LEADER
I20260926 11:39:35.746853 19114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } }
I20260926 11:39:35.747421 19109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:35.748878 19116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a412a2b0c0d49cab1b5d10dd91e2f5f. Latest consensus state: current_term: 1 leader_uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } }
I20260926 11:39:35.748935 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a412a2b0c0d49cab1b5d10dd91e2f5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40393 } } }
I20260926 11:39:35.749015 19116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:35.749054 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:35.749451 19124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:35.752972 19124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:35.759913 19124 catalog_manager.cc:1347] Generated new cluster ID: 0b03eafde90046ff82d7aa2f936e6212
I20260926 11:39:35.760004 19124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:35.767565 19124 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:35.769193 19124 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:35.778198 19124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f: Generated new TSK 0
I20260926 11:39:35.779044 19124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:35.788302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40393
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:39:35.923843 19133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:35.924175 19133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:35.924226 19133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:35.928556 19133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:35.928633 19133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:35.928694 19133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:39:35.934469 19133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:35.935707 19133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:35.937135 19133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:35.947420 19139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:35.947541 19138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:35.947420 19141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:35.947676 19133 server_base.cc:1034] running on GCE node
I20260926 11:39:35.948329 19133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:35.949146 19133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:35.950352 19133 hybrid_clock.cc:648] HybridClock initialized: now 1790422775950339 us; error 33 us; skew 500 ppm
I20260926 11:39:35.952756 19133 webserver.cc:460] Webserver started at http://127.0.98.1:45305/ using document root <none> and password file <none>
I20260926 11:39:35.953403 19133 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:35.953505 19133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:35.953740 19133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:35.956156 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "a3501542ba71415f81956cd0fc91368d"
format_stamp: "Formatted at 2026-09-26 11:39:35 on dist-test-slave-4pw5"
server_key: "0f05c1adf064175b8c152c56b5e7e5b3"
server_key_iv: "89b71416fad66ce06628d71260cf9b08"
server_key_version: "encryptionkey@0"
I20260926 11:39:35.956823 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a3501542ba71415f81956cd0fc91368d"
format_stamp: "Formatted at 2026-09-26 11:39:35 on dist-test-slave-4pw5"
server_key: "0f05c1adf064175b8c152c56b5e7e5b3"
server_key_iv: "89b71416fad66ce06628d71260cf9b08"
server_key_version: "encryptionkey@0"
I20260926 11:39:35.960939 19133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:35.963676 19147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:35.965121 19133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:35.965327 19133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "a3501542ba71415f81956cd0fc91368d"
format_stamp: "Formatted at 2026-09-26 11:39:35 on dist-test-slave-4pw5"
server_key: "0f05c1adf064175b8c152c56b5e7e5b3"
server_key_iv: "89b71416fad66ce06628d71260cf9b08"
server_key_version: "encryptionkey@0"
I20260926 11:39:35.965469 19133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:35.977494 19133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:35.978291 19133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:35.978514 19133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:35.979153 19133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:35.980266 19133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:35.980341 19133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:35.980418 19133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:35.980461 19133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:35.989959 19133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40085
I20260926 11:39:35.990008 19260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40085 every 8 connection(s)
I20260926 11:39:35.991012 19133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:39:35.996488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 19133
I20260926 11:39:35.996662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:39:35.997031 392 external_mini_cluster.cc:1428] Setting key 252feb87da4e3d71a63f067c9fcdcf99
I20260926 11:39:35.999358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40393
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:39:36.001838 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.993053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42385 (local address 127.0.98.62:40393)
0926 11:39:35.993209 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:35.993214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:35.994358 (+ 1144us) server_negotiation.cc:407] Connection header received
0926 11:39:35.995752 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:35.995755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:35.995818 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:35.995916 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:35.996938 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.997674 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.999091 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.999344 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.999431 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.001257 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:36.001261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:36.001263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:36.001277 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:36.001299 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:36.001309 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:36.001416 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.001594 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.001676 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:39:36.002885 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:35.993511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40393 (local address 127.0.98.1:42385)
0926 11:39:35.994151 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:35.994191 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:35.995467 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:35.996084 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:35.996091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:35.996179 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:35.996767 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.996792 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.997848 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.997852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.998952 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:35.998965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:35.999532 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:35.999540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:35.999832 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.000902 (+ 1070us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:39:36.000904 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:36.000955 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:39:36.001038 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:36.001421 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:36.001434 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:36.001441 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:39:36.001596 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 11:39:36.001984 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
I20260926 11:39:36.004297 19261 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40393
I20260926 11:39:36.004571 19261 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:36.005168 19261 heartbeater.cc:511] Master 127.0.98.62:40393 requested a full tablet report, sending...
I20260926 11:39:36.007491 19073 ts_manager.cc:194] Registered new tserver with Master: a3501542ba71415f81956cd0fc91368d (127.0.98.1:40085)
I20260926 11:39:36.009227 19073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42385
W20260926 11:39:36.121716 19264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:36.122041 19264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:36.122082 19264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:36.126473 19264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:36.126549 19264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:36.126609 19264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:39:36.132519 19264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:36.133839 19264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:36.135421 19264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:36.146155 19272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:36.146075 19270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:36.146092 19264 server_base.cc:1034] running on GCE node
W20260926 11:39:36.146114 19269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:36.146938 19264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:36.147873 19264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:36.149122 19264 hybrid_clock.cc:648] HybridClock initialized: now 1790422776149106 us; error 93 us; skew 500 ppm
I20260926 11:39:36.151314 19264 webserver.cc:460] Webserver started at http://127.0.98.2:40983/ using document root <none> and password file <none>
I20260926 11:39:36.152050 19264 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:36.152163 19264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:36.152385 19264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:36.154704 19264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "46eb4347e8f44edaa28759ff1be2ffab"
format_stamp: "Formatted at 2026-09-26 11:39:36 on dist-test-slave-4pw5"
server_key: "0869a828d14b83e1b7a00c06f3bb9fce"
server_key_iv: "8e15cc838971738bf955e521eecae56a"
server_key_version: "encryptionkey@0"
I20260926 11:39:36.155378 19264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "46eb4347e8f44edaa28759ff1be2ffab"
format_stamp: "Formatted at 2026-09-26 11:39:36 on dist-test-slave-4pw5"
server_key: "0869a828d14b83e1b7a00c06f3bb9fce"
server_key_iv: "8e15cc838971738bf955e521eecae56a"
server_key_version: "encryptionkey@0"
I20260926 11:39:36.159519 19264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 11:39:36.162262 19278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:36.163699 19264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:36.163859 19264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "46eb4347e8f44edaa28759ff1be2ffab"
format_stamp: "Formatted at 2026-09-26 11:39:36 on dist-test-slave-4pw5"
server_key: "0869a828d14b83e1b7a00c06f3bb9fce"
server_key_iv: "8e15cc838971738bf955e521eecae56a"
server_key_version: "encryptionkey@0"
I20260926 11:39:36.163993 19264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:36.189296 19264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:36.190207 19264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:36.190446 19264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:36.191180 19264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:36.192378 19264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:36.192458 19264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:36.192543 19264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:36.192589 19264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:36.203418 19264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41045
I20260926 11:39:36.203439 19391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41045 every 8 connection(s)
I20260926 11:39:36.204478 19264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:39:36.207705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 19264
I20260926 11:39:36.207847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:39:36.208199 392 external_mini_cluster.cc:1428] Setting key 22438202fb61a9cb9d8a262cd991b5e4
I20260926 11:39:36.210448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40393
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:39:36.216187 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.206742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38869 (local address 127.0.98.62:40393)
0926 11:39:36.206924 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:36.206928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:36.208151 (+ 1223us) server_negotiation.cc:407] Connection header received
0926 11:39:36.209825 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:36.209828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:36.209884 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:36.209972 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:36.211008 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.211919 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.213629 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.214080 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.214213 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.215507 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:36.215513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:36.215519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:36.215541 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:36.215601 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:36.215614 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:36.215715 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.215895 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.215980 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 11:39:36.217043 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.207144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40393 (local address 127.0.98.2:38869)
0926 11:39:36.207959 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:36.208016 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:36.209572 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:36.210150 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:36.210157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:36.210250 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:36.210857 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.210874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.212076 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.212082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.213486 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.213497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.214258 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.214261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.214418 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.215186 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:39:36.215188 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:36.215240 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:39:36.215321 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:36.215715 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:36.215731 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:36.215737 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:39:36.215847 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 11:39:36.216104 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
I20260926 11:39:36.218374 19392 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40393
I20260926 11:39:36.218639 19392 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:36.219224 19392 heartbeater.cc:511] Master 127.0.98.62:40393 requested a full tablet report, sending...
I20260926 11:39:36.220921 19073 ts_manager.cc:194] Registered new tserver with Master: 46eb4347e8f44edaa28759ff1be2ffab (127.0.98.2:41045)
I20260926 11:39:36.221585 19073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38869
W20260926 11:39:36.330070 19395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:36.330423 19395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:36.330468 19395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:36.335202 19395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:36.335302 19395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:36.335366 19395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:39:36.341025 19395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:36.342478 19395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:36.344077 19395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:36.354930 19403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:36.354969 19400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:36.354930 19401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:36.355386 19395 server_base.cc:1034] running on GCE node
I20260926 11:39:36.355947 19395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:36.356995 19395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:36.358239 19395 hybrid_clock.cc:648] HybridClock initialized: now 1790422776358202 us; error 66 us; skew 500 ppm
I20260926 11:39:36.360875 19395 webserver.cc:460] Webserver started at http://127.0.98.3:38441/ using document root <none> and password file <none>
I20260926 11:39:36.361583 19395 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:36.361660 19395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:36.361905 19395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:36.364638 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "3520a3f7fab44e9a922fd8a1261e4c78"
format_stamp: "Formatted at 2026-09-26 11:39:36 on dist-test-slave-4pw5"
server_key: "2ef220048472123eac742b7d4ec768cb"
server_key_iv: "0bdf1ea6ca5bdcc3455165116d8838ee"
server_key_version: "encryptionkey@0"
I20260926 11:39:36.365434 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3520a3f7fab44e9a922fd8a1261e4c78"
format_stamp: "Formatted at 2026-09-26 11:39:36 on dist-test-slave-4pw5"
server_key: "2ef220048472123eac742b7d4ec768cb"
server_key_iv: "0bdf1ea6ca5bdcc3455165116d8838ee"
server_key_version: "encryptionkey@0"
I20260926 11:39:36.370365 19395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:39:36.373363 19409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:36.374859 19395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 11:39:36.375015 19395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "3520a3f7fab44e9a922fd8a1261e4c78"
format_stamp: "Formatted at 2026-09-26 11:39:36 on dist-test-slave-4pw5"
server_key: "2ef220048472123eac742b7d4ec768cb"
server_key_iv: "0bdf1ea6ca5bdcc3455165116d8838ee"
server_key_version: "encryptionkey@0"
I20260926 11:39:36.375190 19395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:36.385609 19395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:36.386363 19395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:36.386571 19395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:36.387178 19395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:36.388242 19395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:36.388314 19395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:36.388391 19395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:36.388434 19395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:36.397701 19395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42577
I20260926 11:39:36.397739 19522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42577 every 8 connection(s)
I20260926 11:39:36.398699 19395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:39:36.408345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 19395
I20260926 11:39:36.408501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:39:36.408911 392 external_mini_cluster.cc:1428] Setting key 04d80a2eae583814865e015764ed42e1
I20260926 11:39:36.409498 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.400941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47651 (local address 127.0.98.62:40393)
0926 11:39:36.401150 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:36.401155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:36.402301 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 11:39:36.403971 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:36.403975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:36.404039 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:36.404159 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:36.405157 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.405711 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.406968 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.407230 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.407356 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.408741 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:36.408745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:36.408749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:36.408768 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:36.408797 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:36.408809 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:36.408913 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.409269 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.409350 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 11:39:36.410655 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.401394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40393 (local address 127.0.98.3:47651)
0926 11:39:36.402126 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:36.402169 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:36.403721 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:36.404350 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:36.404357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:36.404448 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:36.404981 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.404994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.405881 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.405886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.406819 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.406830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.407409 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.407413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.407657 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.408390 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:39:36.408392 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:36.408436 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:39:36.408513 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:36.408942 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:36.408955 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:36.408960 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:39:36.409198 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 11:39:36.409588 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20260926 11:39:36.411875 19523 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40393
I20260926 11:39:36.412153 19523 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:36.412730 19523 heartbeater.cc:511] Master 127.0.98.62:40393 requested a full tablet report, sending...
I20260926 11:39:36.414132 19073 ts_manager.cc:194] Registered new tserver with Master: 3520a3f7fab44e9a922fd8a1261e4c78 (127.0.98.3:42577)
I20260926 11:39:36.414780 19073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47651
I20260926 11:39:36.423763 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:39:36.430989 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.426482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42352 (local address 127.0.98.62:40393)
0926 11:39:36.426622 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:36.426626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:36.426794 (+ 168us) server_negotiation.cc:407] Connection header received
0926 11:39:36.426848 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:36.426850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:36.426898 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:36.426978 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:36.427277 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.427907 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.428987 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.429269 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.429379 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.430357 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:36.430361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:36.430363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:36.430378 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:36.430399 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:36.430408 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:36.430523 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.430819 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.430865 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:39:36.435226 19073 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:39:36.438498 19073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:39:36.456035 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.449869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.98.2:41045)
0926 11:39:36.450813 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:36.450827 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:36.450850 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:39:36.450927 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:36.450931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:36.451092 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:36.451250 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:36.451608 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.452832 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.454499 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.455198 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.455347 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.455584 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.455725 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.455839 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":130,"threads_started":1}
I20260926 11:39:36.456029 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.449151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41045 (local address 127.0.0.1:55702)
0926 11:39:36.449806 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:36.449868 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:36.450006 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:36.451256 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:36.451260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:36.451289 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:36.451376 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.451382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.452978 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.452982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.454357 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.454368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.455346 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.455350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.455469 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.455525 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 11:39:36.455689 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 11:39:36.455839 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":40,"thread_start_us":94,"threads_started":1}
I20260926 11:39:36.457054 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.450086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40085 (local address 127.0.0.1:53904)
0926 11:39:36.451474 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:36.451486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:36.451632 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:36.452207 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:36.452210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:36.452228 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:36.452315 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.452325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.453431 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.453437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.454565 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.454576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.456037 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.456040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.456647 (+ 607us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.456690 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 11:39:36.456809 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 11:39:36.456895 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1318,"mutex_wait_us":22,"thread_start_us":95,"threads_started":1}
I20260926 11:39:36.457216 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.450491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53904 (local address 127.0.98.1:40085)
0926 11:39:36.451153 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:36.451167 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:36.451824 (+ 657us) server_negotiation.cc:407] Connection header received
0926 11:39:36.451908 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:36.451913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:36.452060 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:36.452216 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:36.452411 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.453298 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.454702 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.455713 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.455809 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.455973 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.456898 (+ 925us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.457038 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":249,"threads_started":1}
I20260926 11:39:36.460245 19326 tablet_service.cc:1467] Processing CreateTablet for tablet 91382ad7c3f245e6898961b3d7998db3 (DEFAULT_TABLE table=test-table [id=14c3c97039934416a3273e11b10faa85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:36.460238 19195 tablet_service.cc:1467] Processing CreateTablet for tablet 91382ad7c3f245e6898961b3d7998db3 (DEFAULT_TABLE table=test-table [id=14c3c97039934416a3273e11b10faa85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:36.461623 19195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91382ad7c3f245e6898961b3d7998db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:36.461623 19326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91382ad7c3f245e6898961b3d7998db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:36.463292 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.449389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42577 (local address 127.0.0.1:40228)
0926 11:39:36.450316 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:36.450327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:36.450452 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:36.451163 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:36.451172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:36.451202 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:36.451347 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.451357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.454317 (+ 2960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.454322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.455421 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.455432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.462786 (+ 7354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.462791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.462902 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.462930 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 11:39:36.463006 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 11:39:36.463109 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":100,"threads_started":1}
I20260926 11:39:36.463739 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.449540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.0.98.3:42577)
0926 11:39:36.450004 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:36.450017 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:36.450467 (+ 450us) server_negotiation.cc:407] Connection header received
0926 11:39:36.450561 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:36.450566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:36.450813 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:36.450978 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:36.451511 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.452832 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.455614 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.462574 (+ 6960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.462773 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.463019 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.463079 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.463201 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":118,"threads_started":1}
I20260926 11:39:36.467643 19457 tablet_service.cc:1467] Processing CreateTablet for tablet 91382ad7c3f245e6898961b3d7998db3 (DEFAULT_TABLE table=test-table [id=14c3c97039934416a3273e11b10faa85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:36.469192 19457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91382ad7c3f245e6898961b3d7998db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:36.470831 19540 tablet_bootstrap.cc:492] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: Bootstrap starting.
I20260926 11:39:36.472846 19541 tablet_bootstrap.cc:492] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d: Bootstrap starting.
I20260926 11:39:36.473605 19540 tablet_bootstrap.cc:654] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:36.474596 19540 log.cc:824] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:36.475873 19541 tablet_bootstrap.cc:654] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:36.476706 19540 tablet_bootstrap.cc:492] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: No bootstrap required, opened a new log
I20260926 11:39:36.476845 19541 log.cc:824] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:36.476938 19540 ts_tablet_manager.cc:1397] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:39:36.479328 19541 tablet_bootstrap.cc:492] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d: No bootstrap required, opened a new log
I20260926 11:39:36.479552 19541 ts_tablet_manager.cc:1397] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 11:39:36.479781 19544 tablet_bootstrap.cc:492] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: Bootstrap starting.
I20260926 11:39:36.481257 19540 raft_consensus.cc:348] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } }
I20260926 11:39:36.481590 19540 raft_consensus.cc:374] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:36.481652 19540 raft_consensus.cc:729] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46eb4347e8f44edaa28759ff1be2ffab, State: Initialized, Role: FOLLOWER
I20260926 11:39:36.482127 19540 consensus_queue.cc:260] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } }
I20260926 11:39:36.482893 19392 heartbeater.cc:503] Master 127.0.98.62:40393 was elected leader, sending a full tablet report...
I20260926 11:39:36.483003 19544 tablet_bootstrap.cc:654] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:36.483233 19540 ts_tablet_manager.cc:1428] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:39:36.483932 19544 log.cc:824] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:36.484524 19541 raft_consensus.cc:348] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } }
I20260926 11:39:36.484831 19541 raft_consensus.cc:374] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:36.484903 19541 raft_consensus.cc:729] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3501542ba71415f81956cd0fc91368d, State: Initialized, Role: FOLLOWER
I20260926 11:39:36.485435 19541 consensus_queue.cc:260] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } }
I20260926 11:39:36.486171 19261 heartbeater.cc:503] Master 127.0.98.62:40393 was elected leader, sending a full tablet report...
I20260926 11:39:36.486318 19544 tablet_bootstrap.cc:492] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: No bootstrap required, opened a new log
I20260926 11:39:36.486514 19544 ts_tablet_manager.cc:1397] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 11:39:36.486542 19541 ts_tablet_manager.cc:1428] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260926 11:39:36.491037 19544 raft_consensus.cc:348] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } }
I20260926 11:39:36.491355 19544 raft_consensus.cc:374] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:36.491394 19544 raft_consensus.cc:729] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3520a3f7fab44e9a922fd8a1261e4c78, State: Initialized, Role: FOLLOWER
I20260926 11:39:36.491864 19544 consensus_queue.cc:260] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } }
W20260926 11:39:36.492388 19262 tablet.cc:2369] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:39:36.492570 19523 heartbeater.cc:503] Master 127.0.98.62:40393 was elected leader, sending a full tablet report...
I20260926 11:39:36.492982 19544 ts_tablet_manager.cc:1428] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:39:36.524847 19547 raft_consensus.cc:482] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:36.525128 19547 raft_consensus.cc:504] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } }
I20260926 11:39:36.526547 19547 leader_election.cc:288] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46eb4347e8f44edaa28759ff1be2ffab (127.0.98.2:41045), 3520a3f7fab44e9a922fd8a1261e4c78 (127.0.98.3:42577)
I20260926 11:39:36.533138 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.527015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56565 (local address 127.0.98.2:41045)
0926 11:39:36.527260 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:36.527263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:36.527361 (+ 98us) server_negotiation.cc:407] Connection header received
0926 11:39:36.527565 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:36.527590 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:36.527650 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:36.527740 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:36.528305 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.529257 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.531477 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.532192 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.532328 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.532400 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.532881 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.532983 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 11:39:36.533183 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.527015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33881 (local address 127.0.98.3:42577)
0926 11:39:36.527160 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:36.527165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:36.527500 (+ 335us) server_negotiation.cc:407] Connection header received
0926 11:39:36.527612 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:36.527616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:36.527664 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:36.527745 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:36.528305 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.529241 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.531477 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.532192 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.532328 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.532400 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.532882 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.532992 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:39:36.533177 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.526877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41045 (local address 127.0.98.1:56565)
0926 11:39:36.527246 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:36.527263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:36.527429 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:36.527791 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:36.527794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:36.527841 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:36.528110 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.528119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.529414 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.529418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.531280 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.531296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.532382 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.532387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.532619 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.532689 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 11:39:36.532787 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 11:39:36.532952 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":322,"spinlock_wait_cycles":7936,"thread_start_us":104,"threads_started":1}
I20260926 11:39:36.533190 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.526877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42577 (local address 127.0.98.1:33881)
0926 11:39:36.527383 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:36.527397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:36.527488 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:36.527791 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:36.527794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:36.527840 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:36.528110 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.528119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.529414 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.529418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.531280 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:36.531296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.532382 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.532387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:36.532588 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.532689 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 11:39:36.532787 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 11:39:36.532952 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":419,"spinlock_wait_cycles":34560,"thread_start_us":86,"threads_started":1}
I20260926 11:39:36.534034 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91382ad7c3f245e6898961b3d7998db3" candidate_uuid: "a3501542ba71415f81956cd0fc91368d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46eb4347e8f44edaa28759ff1be2ffab" is_pre_election: true
I20260926 11:39:36.534365 19477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91382ad7c3f245e6898961b3d7998db3" candidate_uuid: "a3501542ba71415f81956cd0fc91368d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" is_pre_election: true
I20260926 11:39:36.534423 19346 raft_consensus.cc:2454] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3501542ba71415f81956cd0fc91368d in term 0.
I20260926 11:39:36.534657 19477 raft_consensus.cc:2454] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3501542ba71415f81956cd0fc91368d in term 0.
I20260926 11:39:36.534994 19150 leader_election.cc:302] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46eb4347e8f44edaa28759ff1be2ffab, a3501542ba71415f81956cd0fc91368d; no voters:
I20260926 11:39:36.535331 19547 raft_consensus.cc:2781] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:39:36.535429 19547 raft_consensus.cc:482] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:39:36.535480 19547 raft_consensus.cc:3037] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:36.536881 19547 raft_consensus.cc:504] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } }
I20260926 11:39:36.537374 19547 leader_election.cc:288] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [CANDIDATE]: Term 1 election: Requested vote from peers 46eb4347e8f44edaa28759ff1be2ffab (127.0.98.2:41045), 3520a3f7fab44e9a922fd8a1261e4c78 (127.0.98.3:42577)
I20260926 11:39:36.537819 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91382ad7c3f245e6898961b3d7998db3" candidate_uuid: "a3501542ba71415f81956cd0fc91368d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46eb4347e8f44edaa28759ff1be2ffab"
I20260926 11:39:36.537954 19346 raft_consensus.cc:3037] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:36.537976 19477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91382ad7c3f245e6898961b3d7998db3" candidate_uuid: "a3501542ba71415f81956cd0fc91368d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3520a3f7fab44e9a922fd8a1261e4c78"
I20260926 11:39:36.538239 19477 raft_consensus.cc:3037] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:36.539052 19346 raft_consensus.cc:2454] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3501542ba71415f81956cd0fc91368d in term 1.
I20260926 11:39:36.539484 19150 leader_election.cc:302] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46eb4347e8f44edaa28759ff1be2ffab, a3501542ba71415f81956cd0fc91368d; no voters:
I20260926 11:39:36.539507 19477 raft_consensus.cc:2454] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3501542ba71415f81956cd0fc91368d in term 1.
I20260926 11:39:36.539777 19547 raft_consensus.cc:2781] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:36.540113 19547 raft_consensus.cc:686] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 1 LEADER]: Becoming Leader. State: Replica: a3501542ba71415f81956cd0fc91368d, State: Running, Role: LEADER
I20260926 11:39:36.540513 19547 consensus_queue.cc:237] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } }
I20260926 11:39:36.544282 19073 catalog_manager.cc:5495] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d reported cstate change: term changed from 0 to 1, leader changed from <none> to a3501542ba71415f81956cd0fc91368d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a3501542ba71415f81956cd0fc91368d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3501542ba71415f81956cd0fc91368d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40085 } health_report { overall_health: HEALTHY } } }
I20260926 11:39:36.559650 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.555763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.0.98.1:40085)
0926 11:39:36.555903 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:36.555908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:36.555924 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:39:36.556056 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:36.556059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:36.556122 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:36.556245 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:39:36.556508 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.557035 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.557989 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:36.558228 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:36.558343 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:36.558704 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:39:36.559139 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:39:36.559304 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:36.559392 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 11:39:36.559432 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:39:36.566455 19346 raft_consensus.cc:1261] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [term 1 FOLLOWER]: Refusing update from remote peer a3501542ba71415f81956cd0fc91368d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:36.566499 19477 raft_consensus.cc:1261] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [term 1 FOLLOWER]: Refusing update from remote peer a3501542ba71415f81956cd0fc91368d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:36.567337 19547 consensus_queue.cc:1035] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3520a3f7fab44e9a922fd8a1261e4c78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:36.567673 19551 consensus_queue.cc:1035] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [LEADER]: Connected to new peer: Peer: permanent_uuid: "46eb4347e8f44edaa28759ff1be2ffab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:36.577677 19556 mvcc.cc:205] Tried to move back new op lower bound from 7333571692807880704 to 7333571692713009152. Current Snapshot: MvccSnapshot[applied={T|T < 7333571692807880704}]
I20260926 11:39:36.587529 19073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:36.587790 19073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:36.590849 19073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a412a2b0c0d49cab1b5d10dd91e2f5f: Sending DeleteTablet for 3 replicas of tablet 91382ad7c3f245e6898961b3d7998db3
I20260926 11:39:36.591982 19457 tablet_service.cc:1514] Processing DeleteTablet for tablet 91382ad7c3f245e6898961b3d7998db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:36 UTC) from {username='slave'} at 127.0.0.1:40228
I20260926 11:39:36.591986 19326 tablet_service.cc:1514] Processing DeleteTablet for tablet 91382ad7c3f245e6898961b3d7998db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:36 UTC) from {username='slave'} at 127.0.0.1:55702
I20260926 11:39:36.592474 19195 tablet_service.cc:1514] Processing DeleteTablet for tablet 91382ad7c3f245e6898961b3d7998db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:36 UTC) from {username='slave'} at 127.0.0.1:53904
I20260926 11:39:36.592875 19565 tablet_replica.cc:330] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: stopping tablet replica
I20260926 11:39:36.593057 19566 tablet_replica.cc:330] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d: stopping tablet replica
I20260926 11:39:36.593398 19566 raft_consensus.cc:2229] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:39:36.593393 19565 raft_consensus.cc:2229] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:36.593819 19566 raft_consensus.cc:2258] T 91382ad7c3f245e6898961b3d7998db3 P a3501542ba71415f81956cd0fc91368d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:36.593878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 19133
I20260926 11:39:36.595803 19565 raft_consensus.cc:2258] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:36.596292 19564 tablet_replica.cc:330] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: stopping tablet replica
I20260926 11:39:36.596661 19564 raft_consensus.cc:2229] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:36.596998 19564 raft_consensus.cc:2258] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:36.597714 19565 ts_tablet_manager.cc:1905] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:36.598347 19564 ts_tablet_manager.cc:1905] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:36.600632 19564 ts_tablet_manager.cc:1918] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:36.600768 19564 log.cc:1196] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/91382ad7c3f245e6898961b3d7998db3
I20260926 11:39:36.601052 19564 ts_tablet_manager.cc:1939] T 91382ad7c3f245e6898961b3d7998db3 P 3520a3f7fab44e9a922fd8a1261e4c78: Deleting consensus metadata
I20260926 11:39:36.601708 19565 ts_tablet_manager.cc:1918] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:36.601830 19565 log.cc:1196] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/91382ad7c3f245e6898961b3d7998db3
I20260926 11:39:36.602190 19565 ts_tablet_manager.cc:1939] T 91382ad7c3f245e6898961b3d7998db3 P 46eb4347e8f44edaa28759ff1be2ffab: Deleting consensus metadata
I20260926 11:39:36.602535 19058 catalog_manager.cc:4841] TS 3520a3f7fab44e9a922fd8a1261e4c78 (127.0.98.3:42577): tablet 91382ad7c3f245e6898961b3d7998db3 (table test-table [id=14c3c97039934416a3273e11b10faa85]) successfully deleted
W20260926 11:39:36.602749 19057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40085 (error 108)
I20260926 11:39:36.603061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 19264
I20260926 11:39:36.604545 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.604140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40085 (local address 127.0.0.1:53916)
0926 11:39:36.604331 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:36.604425 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 11:39:36.604825 19057 catalog_manager.cc:4584] TS a3501542ba71415f81956cd0fc91368d (127.0.98.1:40085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91382ad7c3f245e6898961b3d7998db3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40085: connect: Connection refused (error 111)
I20260926 11:39:36.609683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 19395
I20260926 11:39:36.611505 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:36.609841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41045 (local address 127.0.0.1:55718)
0926 11:39:36.611331 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:36.611410 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1390}
I20260926 11:39:36.616410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 19042
2026-09-26T11:39:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1125 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:36 dist-test-slave-4pw5 krb5kdc[19570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:36 dist-test-slave-4pw5 krb5kdc[19570](info): set up 2 sockets
Sep 26 11:39:36 dist-test-slave-4pw5 krb5kdc[19570](info): commencing operation
krb5kdc: starting...
W20260926 11:39:37.715831 392 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:37 dist-test-slave-4pw5 krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:39:37.852734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:37.970631 19583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:37.970997 19583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:37.971091 19583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:37.975361 19583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:37.975483 19583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:37.975519 19583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:37.975558 19583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:37.975610 19583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:37.981835 19583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:37.983078 19583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:37.984284 19583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:37.993312 19588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:37.993335 19591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:37.993392 19583 server_base.cc:1034] running on GCE node
W20260926 11:39:37.993327 19589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:37.994252 19583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:37.995388 19583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:37.996577 19583 hybrid_clock.cc:648] HybridClock initialized: now 1790422777996552 us; error 53 us; skew 500 ppm
Sep 26 11:39:37 dist-test-slave-4pw5 krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:37.999656 19583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:39:38.000654 19583 webserver.cc:460] Webserver started at http://127.0.98.62:39403/ using document root <none> and password file <none>
I20260926 11:39:38.001291 19583 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:38.001390 19583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:38.001636 19583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:38.004364 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "3d0e182446304e3eb247331c50c35064"
format_stamp: "Formatted at 2026-09-26 11:39:37 on dist-test-slave-4pw5"
server_key: "f8aa7b6dd21360dabf5d55e7c1e08e80"
server_key_iv: "b9aa679efd00aa63c2cdfe6f10d07ea6"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.005244 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d0e182446304e3eb247331c50c35064"
format_stamp: "Formatted at 2026-09-26 11:39:37 on dist-test-slave-4pw5"
server_key: "f8aa7b6dd21360dabf5d55e7c1e08e80"
server_key_iv: "b9aa679efd00aa63c2cdfe6f10d07ea6"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.009956 19583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:39:38.013123 19598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.015282 19583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 11:39:38.015465 19583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "3d0e182446304e3eb247331c50c35064"
format_stamp: "Formatted at 2026-09-26 11:39:37 on dist-test-slave-4pw5"
server_key: "f8aa7b6dd21360dabf5d55e7c1e08e80"
server_key_iv: "b9aa679efd00aa63c2cdfe6f10d07ea6"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.015671 19583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:38.036814 19583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:38.040436 19583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:38.040674 19583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:38.047947 19583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37145
I20260926 11:39:38.048019 19650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37145 every 8 connection(s)
I20260926 11:39:38.049217 19583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:38.050055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 19583
I20260926 11:39:38.050203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:38.050537 392 external_mini_cluster.cc:1428] Setting key d2805147f8394af095777fcdebcaa4aa
I20260926 11:39:38.052845 19651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:38.059813 19651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064: Bootstrap starting.
Sep 26 11:39:38 dist-test-slave-4pw5 krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:39:38.062762 19651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:38.063743 19651 log.cc:824] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:38.066365 19651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064: No bootstrap required, opened a new log
I20260926 11:39:38.066883 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.052163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.0.98.62:37145)
0926 11:39:38.052682 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:38.052701 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:38.052739 (+ 38us) server_negotiation.cc:407] Connection header received
0926 11:39:38.053479 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:38.053498 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:38.053845 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:38.054130 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:38.055119 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.056294 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.057472 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.057930 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.058048 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.061126 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:38.061137 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:38.061159 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:38.061212 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:38.063325 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.063864 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.063870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.063876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.064016 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.064339 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.064342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.064343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.064874 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:38.065061 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:38.065513 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 11:39:38.065888 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":157,"threads_started":1}
I20260926 11:39:38.070629 19651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0e182446304e3eb247331c50c35064" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37145 } }
I20260926 11:39:38.070884 19651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:38.070922 19651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0e182446304e3eb247331c50c35064, State: Initialized, Role: FOLLOWER
I20260926 11:39:38.071318 19651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0e182446304e3eb247331c50c35064" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37145 } }
I20260926 11:39:38.071435 19651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:38.071475 19651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:38.071550 19651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:38.072628 19651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0e182446304e3eb247331c50c35064" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37145 } }
I20260926 11:39:38.072976 19651 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0e182446304e3eb247331c50c35064; no voters:
I20260926 11:39:38.073392 19651 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:38.073601 19656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:38.073885 19656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0e182446304e3eb247331c50c35064, State: Running, Role: LEADER
I20260926 11:39:38.074252 19656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0e182446304e3eb247331c50c35064" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37145 } }
I20260926 11:39:38.074973 19651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:38.076865 19658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0e182446304e3eb247331c50c35064. Latest consensus state: current_term: 1 leader_uuid: "3d0e182446304e3eb247331c50c35064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0e182446304e3eb247331c50c35064" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37145 } } }
I20260926 11:39:38.077080 19658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:38.077445 19657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0e182446304e3eb247331c50c35064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0e182446304e3eb247331c50c35064" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37145 } } }
I20260926 11:39:38.077545 19657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:38.077656 19666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:38.080905 19666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:38.087522 19666 catalog_manager.cc:1347] Generated new cluster ID: a23e739670ef49768b18d967abfb1f02
I20260926 11:39:38.087668 19666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:38.114089 19666 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:38.115494 19666 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:38.124038 19666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064: Generated new TSK 0
I20260926 11:39:38.124997 19666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:39:38.168453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37145
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:38.286216 19677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:38.286582 19677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:38.286679 19677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:38.291100 19677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:38.291210 19677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:38.291311 19677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:39:38.297058 19677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:38.298292 19677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:38.299363 19677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:38.309695 19685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:38.309768 19682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:38.309768 19683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:38.309902 19677 server_base.cc:1034] running on GCE node
I20260926 11:39:38.310420 19677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:38.311173 19677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:38.312420 19677 hybrid_clock.cc:648] HybridClock initialized: now 1790422778312408 us; error 32 us; skew 500 ppm
Sep 26 11:39:38 dist-test-slave-4pw5 krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:38.315491 19677 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:39:38.316493 19677 webserver.cc:460] Webserver started at http://127.0.98.1:41797/ using document root <none> and password file <none>
I20260926 11:39:38.317116 19677 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:38.317221 19677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:38.317464 19677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:38.319857 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "12b3ce1f03bd4ea383f5e365a2b3e546"
format_stamp: "Formatted at 2026-09-26 11:39:38 on dist-test-slave-4pw5"
server_key: "1873aac9072d7741f91cce0a19c2127f"
server_key_iv: "7980a79d0d1cbc58d997da26dc9e9bd0"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.320523 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "12b3ce1f03bd4ea383f5e365a2b3e546"
format_stamp: "Formatted at 2026-09-26 11:39:38 on dist-test-slave-4pw5"
server_key: "1873aac9072d7741f91cce0a19c2127f"
server_key_iv: "7980a79d0d1cbc58d997da26dc9e9bd0"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.324774 19677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:38.327528 19692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.328966 19677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:38.329077 19677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "12b3ce1f03bd4ea383f5e365a2b3e546"
format_stamp: "Formatted at 2026-09-26 11:39:38 on dist-test-slave-4pw5"
server_key: "1873aac9072d7741f91cce0a19c2127f"
server_key_iv: "7980a79d0d1cbc58d997da26dc9e9bd0"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.329241 19677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:38.340866 19677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:38.344329 19677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:38.344547 19677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:38.345227 19677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:38.346175 19677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:38.346232 19677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.346273 19677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:38.346328 19677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.355458 19677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32899
I20260926 11:39:38.355499 19805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32899 every 8 connection(s)
I20260926 11:39:38.356550 19677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:39:38.366405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 19677
I20260926 11:39:38.366519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:39:38.366883 392 external_mini_cluster.cc:1428] Setting key 325980e32d075d6bd336e42033e83855
Sep 26 11:39:38 dist-test-slave-4pw5 krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:39:38.373098 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.358698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46707 (local address 127.0.98.62:37145)
0926 11:39:38.358882 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:38.358886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:38.359882 (+ 996us) server_negotiation.cc:407] Connection header received
0926 11:39:38.361352 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:38.361357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:38.361423 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:38.361542 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:38.362835 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.363400 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.364708 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.364958 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.365086 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.367996 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:38.367999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:38.368002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:38.368029 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:38.370091 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.370862 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.370865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.370866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.370949 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.371495 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.371498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.371499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.371845 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:38.371934 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:38.372633 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 11:39:38.372833 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 11:39:38.373775 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.359067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37145 (local address 127.0.98.1:46707)
0926 11:39:38.359704 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:38.359756 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:38.361104 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:38.361729 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:38.361737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:38.362148 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:38.362658 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.362673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.363549 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.363553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.364563 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.364575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.365142 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.365146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.365333 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.366038 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:38.366056 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:38.367813 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:38.370255 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:38.370264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:38.370283 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:38.370709 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:38.371108 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:38.371113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:38.371116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:38.371348 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:38.371964 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:38.371974 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:38.372347 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 11:39:38.372512 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 11:39:38.372823 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":140,"threads_started":1}
I20260926 11:39:38.375128 19806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37145
I20260926 11:39:38.375422 19806 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:38.375986 19806 heartbeater.cc:511] Master 127.0.98.62:37145 requested a full tablet report, sending...
I20260926 11:39:38.377802 19615 ts_manager.cc:194] Registered new tserver with Master: 12b3ce1f03bd4ea383f5e365a2b3e546 (127.0.98.1:32899)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:39:38.378959 19615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:39:38.399830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37145
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:38.520637 19811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:38.520934 19811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:38.520980 19811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:38.525602 19811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:38.525738 19811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:38.525879 19811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:39:38.532677 19811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:38.533962 19811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:38.535082 19811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:38.546000 19819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:38.546134 19811 server_base.cc:1034] running on GCE node
W20260926 11:39:38.546002 19816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:38.545987 19817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:38.546842 19811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:38.547637 19811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:38.548857 19811 hybrid_clock.cc:648] HybridClock initialized: now 1790422778548835 us; error 38 us; skew 500 ppm
Sep 26 11:39:38 dist-test-slave-4pw5 krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:38.552726 19811 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:39:38.553691 19811 webserver.cc:460] Webserver started at http://127.0.98.2:39871/ using document root <none> and password file <none>
I20260926 11:39:38.554251 19811 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:38.554308 19811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:38.554476 19811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:38.556654 19811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "ff768ac5cdd7468cbd50b90b3e08c002"
format_stamp: "Formatted at 2026-09-26 11:39:38 on dist-test-slave-4pw5"
server_key: "a8eb53092562e9e3eb53db92dba0cfb2"
server_key_iv: "9d943517da154d56a3828c6919da32b3"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.557165 19811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ff768ac5cdd7468cbd50b90b3e08c002"
format_stamp: "Formatted at 2026-09-26 11:39:38 on dist-test-slave-4pw5"
server_key: "a8eb53092562e9e3eb53db92dba0cfb2"
server_key_iv: "9d943517da154d56a3828c6919da32b3"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.561169 19811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:38.564177 19826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.565799 19811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:38.565974 19811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "ff768ac5cdd7468cbd50b90b3e08c002"
format_stamp: "Formatted at 2026-09-26 11:39:38 on dist-test-slave-4pw5"
server_key: "a8eb53092562e9e3eb53db92dba0cfb2"
server_key_iv: "9d943517da154d56a3828c6919da32b3"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.566143 19811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:38.589066 19811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:38.593181 19811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:38.593364 19811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:38.593984 19811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:38.594848 19811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:38.594897 19811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.594941 19811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:38.594957 19811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.604751 19811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38045
I20260926 11:39:38.604779 19939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38045 every 8 connection(s)
I20260926 11:39:38.605772 19811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:39:38.607357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 19811
I20260926 11:39:38.607486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:39:38.607841 392 external_mini_cluster.cc:1428] Setting key 82c179230f48c3c9c179f1b8f18ae598
Sep 26 11:39:38 dist-test-slave-4pw5 krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:39:38.622630 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.607804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47171 (local address 127.0.98.62:37145)
0926 11:39:38.607956 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:38.607967 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:38.608973 (+ 1006us) server_negotiation.cc:407] Connection header received
0926 11:39:38.610458 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:38.610462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:38.610528 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:38.610606 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:38.612357 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.612926 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.614880 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.615140 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.615240 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.618459 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:38.618462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:38.618464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:38.618493 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:38.620224 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.620852 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.620855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.620856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.620982 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.621387 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.621391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.621392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.621598 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:38.621719 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:38.622313 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 11:39:38.622456 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:39:38.623626 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.608330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37145 (local address 127.0.98.2:47171)
0926 11:39:38.608833 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:38.608878 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:38.610237 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:38.610807 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:38.610820 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:38.611378 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:38.612172 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.612196 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.613078 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.613084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.614686 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.614706 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.615340 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.615347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.615627 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.616360 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:38.616388 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:38.618266 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:38.620372 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:38.620378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:38.620392 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:38.620742 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:38.621084 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:38.621087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:38.621089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:38.621272 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:38.621711 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:38.621719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:38.622016 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 11:39:38.622249 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 11:39:38.622635 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260926 11:39:38.625003 19940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37145
I20260926 11:39:38.625308 19940 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:38.625999 19940 heartbeater.cc:511] Master 127.0.98.62:37145 requested a full tablet report, sending...
I20260926 11:39:38.627278 19615 ts_manager.cc:194] Registered new tserver with Master: ff768ac5cdd7468cbd50b90b3e08c002 (127.0.98.2:38045)
I20260926 11:39:38.627950 19615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:39:38.639493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37145
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:38.758641 19945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:38.758996 19945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:38.759075 19945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:38.763543 19945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:38.763723 19945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:38.763830 19945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:39:38.769732 19945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:38.771064 19945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:38.772220 19945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:38.782791 19950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:38.782802 19951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:38.782797 19953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:38.782876 19945 server_base.cc:1034] running on GCE node
I20260926 11:39:38.783553 19945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:38.784335 19945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:38.785557 19945 hybrid_clock.cc:648] HybridClock initialized: now 1790422778785518 us; error 53 us; skew 500 ppm
Sep 26 11:39:38 dist-test-slave-4pw5 krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:38.788956 19945 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:39:38.789995 19945 webserver.cc:460] Webserver started at http://127.0.98.3:37439/ using document root <none> and password file <none>
I20260926 11:39:38.790675 19945 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:38.790771 19945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:38.790977 19945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:38.793197 19945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ca813d912d6458394ad222b87f5d2b5"
format_stamp: "Formatted at 2026-09-26 11:39:38 on dist-test-slave-4pw5"
server_key: "3f1f8d196550bda391db3fc681fb563b"
server_key_iv: "25b0c5a5a1c5dab4583c8b99c28e9eb7"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.793869 19945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ca813d912d6458394ad222b87f5d2b5"
format_stamp: "Formatted at 2026-09-26 11:39:38 on dist-test-slave-4pw5"
server_key: "3f1f8d196550bda391db3fc681fb563b"
server_key_iv: "25b0c5a5a1c5dab4583c8b99c28e9eb7"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.798048 19945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:39:38.800860 19960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.802332 19945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:38.802479 19945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "5ca813d912d6458394ad222b87f5d2b5"
format_stamp: "Formatted at 2026-09-26 11:39:38 on dist-test-slave-4pw5"
server_key: "3f1f8d196550bda391db3fc681fb563b"
server_key_iv: "25b0c5a5a1c5dab4583c8b99c28e9eb7"
server_key_version: "encryptionkey@0"
I20260926 11:39:38.802605 19945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:38.827791 19945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:38.831072 19945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:38.831297 19945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:38.831962 19945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:38.832947 19945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:38.832995 19945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.833070 19945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:38.833109 19945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:38.845325 19945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35981
I20260926 11:39:38.845345 20073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35981 every 8 connection(s)
I20260926 11:39:38.846326 19945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:39:38.848666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 19945
I20260926 11:39:38.848803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:39:38.849103 392 external_mini_cluster.cc:1428] Setting key 1535a7334f7a9789bbf115ecabd17c11
Sep 26 11:39:38 dist-test-slave-4pw5 krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:39:38.862509 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.848487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54407 (local address 127.0.98.62:37145)
0926 11:39:38.848681 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:38.848688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:38.849826 (+ 1138us) server_negotiation.cc:407] Connection header received
0926 11:39:38.851308 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:38.851313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:38.851376 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:38.851464 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:38.852750 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.853305 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.854543 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.854782 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.854912 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.858114 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:38.858117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:38.858120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:38.858153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:38.859826 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.860688 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.860691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.860692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.860780 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.861258 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.861261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.861262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.861466 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:38.861634 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:38.862198 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 11:39:38.862345 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 11:39:38.863659 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.849063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37145 (local address 127.0.98.3:54407)
0926 11:39:38.849657 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:38.849717 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:38.851074 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:38.851674 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:38.851682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:38.852078 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:38.852586 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.852602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.853450 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.853453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.854384 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.854395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.854962 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.854966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.855175 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.855895 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:39:38.855915 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:38.857895 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:38.859962 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:38.859968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:38.859992 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:38.860544 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:38.860914 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:39:38.860916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:39:38.860918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:39:38.861135 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:39:38.861586 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:38.861592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:38.861880 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 11:39:38.862134 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 11:39:38.862572 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":188,"threads_started":1}
I20260926 11:39:38.864899 20074 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37145
I20260926 11:39:38.865141 20074 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:38.865643 20074 heartbeater.cc:511] Master 127.0.98.62:37145 requested a full tablet report, sending...
I20260926 11:39:38.866891 19615 ts_manager.cc:194] Registered new tserver with Master: 5ca813d912d6458394ad222b87f5d2b5 (127.0.98.3:35981)
I20260926 11:39:38.867476 19615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54407
I20260926 11:39:38.876036 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:39:38.886257 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.878310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.0.98.62:37145)
0926 11:39:38.878487 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:38.878491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:38.878588 (+ 97us) server_negotiation.cc:407] Connection header received
0926 11:39:38.878694 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:38.878697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:38.878751 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:38.878820 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:38.879655 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.880187 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.881152 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.881401 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.881484 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.882913 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:38.882916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:38.882919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:38.882949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:38.884421 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.884886 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.884889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.884890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.884974 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:38.885261 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:38.885264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:38.885265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:38.885508 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:38.885638 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:38.885889 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 11:39:38.886043 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 11:39:38.889678 19615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:39:38.892815 19615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:39:38.908043 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.901602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32899 (local address 127.0.0.1:56684)
0926 11:39:38.902114 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:38.902161 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:38.902315 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:38.903149 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:38.903155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:38.903177 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:38.903340 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.903347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.904682 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.904688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.906271 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.906285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.907478 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.907482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.907612 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.907634 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:39:38.907684 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 11:39:38.907883 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":138,"thread_start_us":85,"threads_started":1}
I20260926 11:39:38.908206 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.902810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35981 (local address 127.0.0.1:48090)
0926 11:39:38.903070 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:38.903083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:38.903156 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:38.904076 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:38.904079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:38.904098 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:38.904189 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.904203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.905501 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.905503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.906577 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.906589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.907800 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.907803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.907894 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.907914 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:39:38.907985 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 11:39:38.908055 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":95,"thread_start_us":89,"threads_started":1}
I20260926 11:39:38.908253 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.901784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.0.98.1:32899)
0926 11:39:38.902619 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:38.902627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:38.902650 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:39:38.902744 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:38.902753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:38.902973 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:38.903492 (+ 519us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:38.903703 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.904575 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.906421 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.907330 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.907821 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.907964 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:38.908015 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 11:39:38.908103 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":132,"threads_started":1}
I20260926 11:39:38.908687 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.902060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.0.98.2:38045)
0926 11:39:38.902486 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:38.902498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:38.902803 (+ 305us) server_negotiation.cc:407] Connection header received
0926 11:39:38.903001 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:38.903007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:38.903192 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:38.903351 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:38.903516 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.904775 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.906694 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.907257 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.907387 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.907546 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:38.907904 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 11:39:38.908006 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":202,"threads_started":1}
I20260926 11:39:38.908824 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.903197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.0.98.3:35981)
0926 11:39:38.903653 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:38.903665 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:38.903685 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:39:38.903755 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:38.903759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:38.903929 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:38.904074 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:38.904321 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.905352 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.906715 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.907470 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.907561 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.907739 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:38.908544 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 11:39:38.908661 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":128,"threads_started":1}
I20260926 11:39:38.909274 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:38.902459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38045 (local address 127.0.0.1:50486)
0926 11:39:38.902713 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:38.902726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:38.902867 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:38.903325 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:38.903328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:38.903341 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:38.903441 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.903447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.904914 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.904918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.906554 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:38.906563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:38.907385 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:38.907389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:38.907504 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:38.907541 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 11:39:38.907694 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 11:39:38.909179 (+ 1485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":1392,"thread_start_us":99,"threads_started":1}
I20260926 11:39:38.911522 19874 tablet_service.cc:1467] Processing CreateTablet for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a (DEFAULT_TABLE table=test-table [id=4f6c8caf7bf442d18eeccb1a0242f847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:38.911538 19740 tablet_service.cc:1467] Processing CreateTablet for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a (DEFAULT_TABLE table=test-table [id=4f6c8caf7bf442d18eeccb1a0242f847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:38.911540 19994 tablet_service.cc:1467] Processing CreateTablet for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a (DEFAULT_TABLE table=test-table [id=4f6c8caf7bf442d18eeccb1a0242f847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:38.912648 19874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:38.912652 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:38.912652 19740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:38.919185 20091 tablet_bootstrap.cc:492] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: Bootstrap starting.
I20260926 11:39:38.920351 20093 tablet_bootstrap.cc:492] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: Bootstrap starting.
I20260926 11:39:38.920532 20092 tablet_bootstrap.cc:492] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546: Bootstrap starting.
I20260926 11:39:38.922107 20091 tablet_bootstrap.cc:654] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:38.922557 20092 tablet_bootstrap.cc:654] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:38.922556 20093 tablet_bootstrap.cc:654] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:38.923286 20091 log.cc:824] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:38.923465 20092 log.cc:824] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:38.923743 20093 log.cc:824] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:38.925558 20092 tablet_bootstrap.cc:492] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546: No bootstrap required, opened a new log
I20260926 11:39:38.925558 20091 tablet_bootstrap.cc:492] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: No bootstrap required, opened a new log
I20260926 11:39:38.925786 20091 ts_tablet_manager.cc:1397] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 11:39:38.925786 20092 ts_tablet_manager.cc:1397] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:39:38.927454 20093 tablet_bootstrap.cc:492] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: No bootstrap required, opened a new log
I20260926 11:39:38.927692 20093 ts_tablet_manager.cc:1397] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260926 11:39:38.929934 20091 raft_consensus.cc:348] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260926 11:39:38.930263 20091 raft_consensus.cc:374] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:38.930327 20091 raft_consensus.cc:729] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff768ac5cdd7468cbd50b90b3e08c002, State: Initialized, Role: FOLLOWER
I20260926 11:39:38.930265 20092 raft_consensus.cc:348] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260926 11:39:38.930495 20092 raft_consensus.cc:374] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:38.930540 20092 raft_consensus.cc:729] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b3ce1f03bd4ea383f5e365a2b3e546, State: Initialized, Role: FOLLOWER
I20260926 11:39:38.930506 20093 raft_consensus.cc:348] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260926 11:39:38.930828 20093 raft_consensus.cc:374] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:38.930806 20091 consensus_queue.cc:260] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260926 11:39:38.930933 20093 raft_consensus.cc:729] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca813d912d6458394ad222b87f5d2b5, State: Initialized, Role: FOLLOWER
I20260926 11:39:38.930908 20092 consensus_queue.cc:260] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260926 11:39:38.931393 20093 consensus_queue.cc:260] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260926 11:39:38.931993 19940 heartbeater.cc:503] Master 127.0.98.62:37145 was elected leader, sending a full tablet report...
I20260926 11:39:38.932056 20092 ts_tablet_manager.cc:1428] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:39:38.932056 20091 ts_tablet_manager.cc:1428] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:39:38.932334 20093 ts_tablet_manager.cc:1428] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:39:38.932623 20074 heartbeater.cc:503] Master 127.0.98.62:37145 was elected leader, sending a full tablet report...
I20260926 11:39:38.932680 19806 heartbeater.cc:503] Master 127.0.98.62:37145 was elected leader, sending a full tablet report...
I20260926 11:39:39.042953 20098 raft_consensus.cc:482] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:39.043256 20098 raft_consensus.cc:504] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260926 11:39:39.044489 20098 leader_election.cc:288] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff768ac5cdd7468cbd50b90b3e08c002 (127.0.98.2:38045), 5ca813d912d6458394ad222b87f5d2b5 (127.0.98.3:35981)
I20260926 11:39:39.048851 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.044890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38045 (local address 127.0.98.1:39305)
0926 11:39:39.045272 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:39.045287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:39.045411 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:39.045750 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:39.045753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:39.045788 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:39.045947 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:39.045953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.046716 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.046721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:39.047562 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:39.047576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.048394 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.048398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:39.048493 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.048518 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 11:39:39.048599 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 11:39:39.048681 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"spinlock_wait_cycles":2432,"thread_start_us":135,"threads_started":1}
I20260926 11:39:39.048856 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.044981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39305 (local address 127.0.98.2:38045)
0926 11:39:39.045145 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:39.045149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:39.045392 (+ 243us) server_negotiation.cc:407] Connection header received
0926 11:39:39.045546 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:39.045550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:39.045628 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:39.045733 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:39.046065 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.046586 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.047733 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.048260 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.048371 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.048435 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:39.048643 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 11:39:39.048722 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:39:39.049378 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.045520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48407 (local address 127.0.98.3:35981)
0926 11:39:39.045686 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:39.045692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:39.045799 (+ 107us) server_negotiation.cc:407] Connection header received
0926 11:39:39.045983 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:39.045986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:39.046039 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:39.046120 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:39.046380 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.046878 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.048099 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.048689 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.048790 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.048847 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:39.049185 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 11:39:39.049275 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:39:39.049830 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.045349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35981 (local address 127.0.98.1:48407)
0926 11:39:39.045686 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:39.045700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:39.045866 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:39.046138 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:39.046141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:39.046157 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:39.046276 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:39.046281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.046989 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.046994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:39.047955 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:39.047966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.048909 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.048913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:39.049020 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.049050 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 11:39:39.049608 (+ 558us) client_negotiation.cc:240] Negotiation successful
0926 11:39:39.049686 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"spinlock_wait_cycles":26112,"thread_start_us":89,"threads_started":1}
I20260926 11:39:39.049799 19894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f1d0fcf6f4b6ab4709ec3ade7dc1a" candidate_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" is_pre_election: true
I20260926 11:39:39.050151 19894 raft_consensus.cc:2454] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12b3ce1f03bd4ea383f5e365a2b3e546 in term 0.
I20260926 11:39:39.050575 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f1d0fcf6f4b6ab4709ec3ade7dc1a" candidate_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ca813d912d6458394ad222b87f5d2b5" is_pre_election: true
I20260926 11:39:39.050853 20012 raft_consensus.cc:2454] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12b3ce1f03bd4ea383f5e365a2b3e546 in term 0.
I20260926 11:39:39.050832 19696 leader_election.cc:302] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b3ce1f03bd4ea383f5e365a2b3e546, ff768ac5cdd7468cbd50b90b3e08c002; no voters:
I20260926 11:39:39.051177 20098 raft_consensus.cc:2781] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:39:39.051278 20098 raft_consensus.cc:482] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:39:39.051317 20098 raft_consensus.cc:3037] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:39.052663 20098 raft_consensus.cc:504] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260926 11:39:39.053068 20098 leader_election.cc:288] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [CANDIDATE]: Term 1 election: Requested vote from peers ff768ac5cdd7468cbd50b90b3e08c002 (127.0.98.2:38045), 5ca813d912d6458394ad222b87f5d2b5 (127.0.98.3:35981)
I20260926 11:39:39.053534 19894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f1d0fcf6f4b6ab4709ec3ade7dc1a" candidate_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff768ac5cdd7468cbd50b90b3e08c002"
I20260926 11:39:39.053705 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f1d0fcf6f4b6ab4709ec3ade7dc1a" candidate_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ca813d912d6458394ad222b87f5d2b5"
I20260926 11:39:39.053697 19894 raft_consensus.cc:3037] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:39.053838 20012 raft_consensus.cc:3037] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:39.055140 19894 raft_consensus.cc:2454] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12b3ce1f03bd4ea383f5e365a2b3e546 in term 1.
I20260926 11:39:39.055135 20012 raft_consensus.cc:2454] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12b3ce1f03bd4ea383f5e365a2b3e546 in term 1.
I20260926 11:39:39.055536 19696 leader_election.cc:302] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b3ce1f03bd4ea383f5e365a2b3e546, 5ca813d912d6458394ad222b87f5d2b5; no voters:
I20260926 11:39:39.055799 20098 raft_consensus.cc:2781] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:39.056037 20098 raft_consensus.cc:686] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 1 LEADER]: Becoming Leader. State: Replica: 12b3ce1f03bd4ea383f5e365a2b3e546, State: Running, Role: LEADER
I20260926 11:39:39.056555 20098 consensus_queue.cc:237] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260926 11:39:39.060357 19614 catalog_manager.cc:5495] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12b3ce1f03bd4ea383f5e365a2b3e546 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b3ce1f03bd4ea383f5e365a2b3e546" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } health_report { overall_health: UNKNOWN } } }
W20260926 11:39:39.097405 20075 tablet.cc:2369] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:39:39.102641 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.098665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.0.98.1:32899)
0926 11:39:39.098829 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:39.098835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:39.098850 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:39:39.098937 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:39.098939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:39.098994 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:39.099118 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:39:39.099394 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.100163 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.101106 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.101341 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.101453 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.101778 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:39:39.102224 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:39:39.102360 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:39.102424 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 11:39:39.102488 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 11:39:39.106999 19941 tablet.cc:2369] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:39:39.107894 19807 tablet.cc:2369] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:39:39.109474 19894 raft_consensus.cc:1261] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [term 1 FOLLOWER]: Refusing update from remote peer 12b3ce1f03bd4ea383f5e365a2b3e546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:39.109474 20012 raft_consensus.cc:1261] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [term 1 FOLLOWER]: Refusing update from remote peer 12b3ce1f03bd4ea383f5e365a2b3e546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:39.110352 20102 consensus_queue.cc:1035] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ca813d912d6458394ad222b87f5d2b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:39.110569 20098 consensus_queue.cc:1035] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff768ac5cdd7468cbd50b90b3e08c002" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:39.122704 20106 mvcc.cc:205] Tried to move back new op lower bound from 7333571703222722560 to 7333571703018790912. Current Snapshot: MvccSnapshot[applied={T|T < 7333571703222722560 or (T in {7333571703222722560})}]
I20260926 11:39:39.128252 19614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:39.128422 19614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:39.131008 19614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d0e182446304e3eb247331c50c35064: Sending DeleteTablet for 3 replicas of tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a
I20260926 11:39:39.131901 19740 tablet_service.cc:1514] Processing DeleteTablet for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56684
I20260926 11:39:39.131901 19874 tablet_service.cc:1514] Processing DeleteTablet for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50486
I20260926 11:39:39.132071 19994 tablet_service.cc:1514] Processing DeleteTablet for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48090
I20260926 11:39:39.132498 20116 tablet_replica.cc:330] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546: stopping tablet replica
I20260926 11:39:39.132861 20116 raft_consensus.cc:2229] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 12b3ce1f03bd4ea383f5e365a2b3e546 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:39:39.133138 20117 tablet_replica.cc:330] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: stopping tablet replica
I20260926 11:39:39.133431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 19677
I20260926 11:39:39.133440 20117 raft_consensus.cc:2229] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:39.133795 20117 raft_consensus.cc:2258] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:39.134231 20115 tablet_replica.cc:330] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: stopping tablet replica
I20260926 11:39:39.134526 20115 raft_consensus.cc:2229] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:39.134835 20115 raft_consensus.cc:2258] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:39.135860 20117 ts_tablet_manager.cc:1905] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:39.136549 20115 ts_tablet_manager.cc:1905] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:39.138839 20115 ts_tablet_manager.cc:1918] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:39.138962 20115 log.cc:1196] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/948f1d0fcf6f4b6ab4709ec3ade7dc1a
I20260926 11:39:39.139236 20115 ts_tablet_manager.cc:1939] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P ff768ac5cdd7468cbd50b90b3e08c002: Deleting consensus metadata
I20260926 11:39:39.140040 20117 ts_tablet_manager.cc:1918] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:39.140156 20117 log.cc:1196] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/948f1d0fcf6f4b6ab4709ec3ade7dc1a
I20260926 11:39:39.140520 20117 ts_tablet_manager.cc:1939] T 948f1d0fcf6f4b6ab4709ec3ade7dc1a P 5ca813d912d6458394ad222b87f5d2b5: Deleting consensus metadata
I20260926 11:39:39.140622 19602 catalog_manager.cc:4841] TS ff768ac5cdd7468cbd50b90b3e08c002 (127.0.98.2:38045): tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a (table test-table [id=4f6c8caf7bf442d18eeccb1a0242f847]) successfully deleted
I20260926 11:39:39.141888 19602 catalog_manager.cc:4841] TS 5ca813d912d6458394ad222b87f5d2b5 (127.0.98.3:35981): tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a (table test-table [id=4f6c8caf7bf442d18eeccb1a0242f847]) successfully deleted
W20260926 11:39:39.142318 19601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32899 (error 108)
I20260926 11:39:39.142589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 19811
I20260926 11:39:39.144380 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.143682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32899 (local address 127.0.0.1:56698)
0926 11:39:39.144184 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:39.144271 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373}
W20260926 11:39:39.144683 19601 catalog_manager.cc:4584] TS 12b3ce1f03bd4ea383f5e365a2b3e546 (127.0.98.1:32899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 948f1d0fcf6f4b6ab4709ec3ade7dc1a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32899: connect: Connection refused (error 111)
I20260926 11:39:39.149178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 19945
I20260926 11:39:39.155385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 19583
2026-09-26T11:39:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2538 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T11:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:39Z Disabled control of system clock
I20260926 11:39:39.186759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44005
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:39:39.306713 20122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:39.307034 20122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:39.307072 20122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:39.311236 20122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:39.311326 20122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:39.311342 20122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:39.311360 20122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:39.311375 20122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:39.316411 20122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:39.317536 20122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:39.318949 20122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:39.327857 20128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:39.327924 20122 server_base.cc:1034] running on GCE node
W20260926 11:39:39.327889 20127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:39.327878 20130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:39.328783 20122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:39.330367 20122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:39.331611 20122 hybrid_clock.cc:648] HybridClock initialized: now 1790422779331539 us; error 53 us; skew 500 ppm
I20260926 11:39:39.335008 20122 webserver.cc:460] Webserver started at http://127.0.98.62:42061/ using document root <none> and password file <none>
I20260926 11:39:39.335711 20122 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:39.335784 20122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:39.336025 20122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:39.338413 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "649e722ad088491b8eab3e0d43f922b3"
format_stamp: "Formatted at 2026-09-26 11:39:39 on dist-test-slave-4pw5"
server_key: "b4e90e699e4e4ab11cc7035d822cbe9d"
server_key_iv: "05cb44835f2cbd625c1b56d9b2d6a66d"
server_key_version: "encryptionkey@0"
I20260926 11:39:39.339037 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "649e722ad088491b8eab3e0d43f922b3"
format_stamp: "Formatted at 2026-09-26 11:39:39 on dist-test-slave-4pw5"
server_key: "b4e90e699e4e4ab11cc7035d822cbe9d"
server_key_iv: "05cb44835f2cbd625c1b56d9b2d6a66d"
server_key_version: "encryptionkey@0"
I20260926 11:39:39.343554 20122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 11:39:39.346244 20136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:39.347770 20122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 11:39:39.347922 20122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "649e722ad088491b8eab3e0d43f922b3"
format_stamp: "Formatted at 2026-09-26 11:39:39 on dist-test-slave-4pw5"
server_key: "b4e90e699e4e4ab11cc7035d822cbe9d"
server_key_iv: "05cb44835f2cbd625c1b56d9b2d6a66d"
server_key_version: "encryptionkey@0"
I20260926 11:39:39.348044 20122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:39.382230 20122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:39.383023 20122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:39.383255 20122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:39.390758 20122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44005
I20260926 11:39:39.390771 20188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44005 every 8 connection(s)
I20260926 11:39:39.391834 20122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:39.394013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 20122
I20260926 11:39:39.394133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:39.394470 392 external_mini_cluster.cc:1428] Setting key 9ec32443b464609b36ed2977a80694b7
I20260926 11:39:39.395612 20189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:39.402467 20189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3: Bootstrap starting.
I20260926 11:39:39.404678 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.396072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.0.98.62:44005)
0926 11:39:39.396756 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:39.396772 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:39.396808 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:39:39.397479 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:39.397497 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:39.397902 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:39.398187 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:39.398574 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.399556 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.400848 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.401347 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.401480 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.402437 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:39.402442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:39.402456 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:39.402987 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:39.403049 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:39.403127 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:39.403216 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:39.403439 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 11:39:39.403845 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":171,"threads_started":1}
I20260926 11:39:39.405045 20189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:39.405838 20189 log.cc:824] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:39.407907 20189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3: No bootstrap required, opened a new log
I20260926 11:39:39.411504 20189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649e722ad088491b8eab3e0d43f922b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44005 } }
I20260926 11:39:39.411875 20189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:39.411973 20189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649e722ad088491b8eab3e0d43f922b3, State: Initialized, Role: FOLLOWER
I20260926 11:39:39.412489 20189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649e722ad088491b8eab3e0d43f922b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44005 } }
I20260926 11:39:39.412648 20189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:39.412719 20189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:39.412844 20189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:39.413885 20189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649e722ad088491b8eab3e0d43f922b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44005 } }
I20260926 11:39:39.414269 20189 leader_election.cc:302] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 649e722ad088491b8eab3e0d43f922b3; no voters:
I20260926 11:39:39.414649 20189 leader_election.cc:288] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:39.414798 20194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:39.415071 20194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [term 1 LEADER]: Becoming Leader. State: Replica: 649e722ad088491b8eab3e0d43f922b3, State: Running, Role: LEADER
I20260926 11:39:39.415398 20194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649e722ad088491b8eab3e0d43f922b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44005 } }
I20260926 11:39:39.416066 20189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:39.417434 20195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "649e722ad088491b8eab3e0d43f922b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649e722ad088491b8eab3e0d43f922b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44005 } } }
I20260926 11:39:39.417634 20195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:39.417433 20196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 649e722ad088491b8eab3e0d43f922b3. Latest consensus state: current_term: 1 leader_uuid: "649e722ad088491b8eab3e0d43f922b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649e722ad088491b8eab3e0d43f922b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44005 } } }
I20260926 11:39:39.417876 20196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:39.418670 20203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:39.422092 20203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:39.428227 20203 catalog_manager.cc:1347] Generated new cluster ID: 2400a75ec2604d8395ed161dbe3d3104
I20260926 11:39:39.428308 20203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:39.436755 20203 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:39.438184 20203 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:39.445969 20203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3: Generated new TSK 0
I20260926 11:39:39.446939 20203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:39.451229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44005
--builtin_ntp_servers=127.0.98.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:39:39.568365 20213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:39.568686 20213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:39.568758 20213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:39.572959 20213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:39.573048 20213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:39.573145 20213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:39:39.578274 20213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:39.579465 20213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:39.580974 20213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:39:39.591342 20213 server_base.cc:1034] running on GCE node
W20260926 11:39:39.591385 20219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:39.591513 20218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:39.591385 20221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:39.592051 20213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:39.592773 20213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:39.593950 20213 hybrid_clock.cc:648] HybridClock initialized: now 1790422779593890 us; error 78 us; skew 500 ppm
I20260926 11:39:39.599855 20213 webserver.cc:460] Webserver started at http://127.0.98.1:46019/ using document root <none> and password file <none>
I20260926 11:39:39.600574 20213 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:39.600652 20213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:39.600891 20213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:39.603108 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/data/instance:
uuid: "259ebb9958cf486b8177dc66a79ae765"
format_stamp: "Formatted at 2026-09-26 11:39:39 on dist-test-slave-4pw5"
server_key: "a65b24f7abd1477e3d4dd0ae61d09ac4"
server_key_iv: "b70c773132991d38aeff91b719cf253c"
server_key_version: "encryptionkey@0"
I20260926 11:39:39.603775 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/wal/instance:
uuid: "259ebb9958cf486b8177dc66a79ae765"
format_stamp: "Formatted at 2026-09-26 11:39:39 on dist-test-slave-4pw5"
server_key: "a65b24f7abd1477e3d4dd0ae61d09ac4"
server_key_iv: "b70c773132991d38aeff91b719cf253c"
server_key_version: "encryptionkey@0"
I20260926 11:39:39.607796 20213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:39.610288 20227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:39.611658 20213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:39:39.611799 20213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/wal
uuid: "259ebb9958cf486b8177dc66a79ae765"
format_stamp: "Formatted at 2026-09-26 11:39:39 on dist-test-slave-4pw5"
server_key: "a65b24f7abd1477e3d4dd0ae61d09ac4"
server_key_iv: "b70c773132991d38aeff91b719cf253c"
server_key_version: "encryptionkey@0"
I20260926 11:39:39.611929 20213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:39.641626 20213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:39.642436 20213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:39.642656 20213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:39.643309 20213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:39.644318 20213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:39.644392 20213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:39.644459 20213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:39.644511 20213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:39.654541 20213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42823
I20260926 11:39:39.654565 20340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42823 every 8 connection(s)
I20260926 11:39:39.655519 20213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:39:39.658785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 20213
I20260926 11:39:39.658939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:39:39.659271 392 external_mini_cluster.cc:1428] Setting key 8c710edd81fb6d541767fa844bfab0ee
I20260926 11:39:39.661684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44005
--builtin_ntp_servers=127.0.98.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:39:39.665967 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.657576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38555 (local address 127.0.98.62:44005)
0926 11:39:39.657738 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:39.657743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:39.658906 (+ 1163us) server_negotiation.cc:407] Connection header received
0926 11:39:39.660774 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:39.660780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:39.660846 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:39.660932 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:39.661946 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.662481 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.663716 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.663972 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.664096 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.665380 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:39.665384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:39.665387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:39.665416 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:39.665436 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:39.665444 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:39.665535 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:39.665724 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 11:39:39.665815 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:39:39.666874 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.657956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44005 (local address 127.0.98.1:38555)
0926 11:39:39.658702 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:39.658762 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:39.660502 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:39.661151 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:39.661158 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:39.661250 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:39.661755 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:39.661767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.662619 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.662621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:39.663531 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:39.663541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.664157 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.664162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:39.664348 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.665030 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:39:39.665031 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:39.665074 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:39:39.665195 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:39.665550 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:39.665561 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:39.665566 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:39:39.665718 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 11:39:39.666037 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
I20260926 11:39:39.668330 20341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44005
I20260926 11:39:39.668622 20341 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:39.669371 20341 heartbeater.cc:511] Master 127.0.98.62:44005 requested a full tablet report, sending...
I20260926 11:39:39.671185 20153 ts_manager.cc:194] Registered new tserver with Master: 259ebb9958cf486b8177dc66a79ae765 (127.0.98.1:42823)
I20260926 11:39:39.672451 20153 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38555
W20260926 11:39:39.779820 20344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:39.780151 20344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:39.780226 20344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:39.784562 20344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:39.784659 20344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:39.784751 20344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:39:39.789750 20344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:39.790896 20344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:39.792336 20344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:39.802933 20350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:39.802933 20352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:39.802937 20349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:39.802979 20344 server_base.cc:1034] running on GCE node
I20260926 11:39:39.803732 20344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:39.804667 20344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:39.805932 20344 hybrid_clock.cc:648] HybridClock initialized: now 1790422779805915 us; error 32 us; skew 500 ppm
I20260926 11:39:39.808270 20344 webserver.cc:460] Webserver started at http://127.0.98.2:33401/ using document root <none> and password file <none>
I20260926 11:39:39.808944 20344 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:39.809047 20344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:39.809304 20344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:39.811628 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/data/instance:
uuid: "95a24b41487d4ab0aadaf47b46bdad7f"
format_stamp: "Formatted at 2026-09-26 11:39:39 on dist-test-slave-4pw5"
server_key: "3cbd1513d09778523c040d36691b327a"
server_key_iv: "cdcf672a058b2adbf724cf91dbefc923"
server_key_version: "encryptionkey@0"
I20260926 11:39:39.812299 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/wal/instance:
uuid: "95a24b41487d4ab0aadaf47b46bdad7f"
format_stamp: "Formatted at 2026-09-26 11:39:39 on dist-test-slave-4pw5"
server_key: "3cbd1513d09778523c040d36691b327a"
server_key_iv: "cdcf672a058b2adbf724cf91dbefc923"
server_key_version: "encryptionkey@0"
I20260926 11:39:39.816752 20344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:39.819772 20358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:39.821153 20344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:39.821321 20344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/wal
uuid: "95a24b41487d4ab0aadaf47b46bdad7f"
format_stamp: "Formatted at 2026-09-26 11:39:39 on dist-test-slave-4pw5"
server_key: "3cbd1513d09778523c040d36691b327a"
server_key_iv: "cdcf672a058b2adbf724cf91dbefc923"
server_key_version: "encryptionkey@0"
I20260926 11:39:39.821465 20344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:39.850304 20344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:39.851085 20344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:39.851312 20344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:39.852005 20344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:39.853101 20344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:39.853176 20344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:39.853246 20344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:39.853276 20344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:39.862607 20344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42009
I20260926 11:39:39.862649 20471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42009 every 8 connection(s)
I20260926 11:39:39.863710 20344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:39:39.869087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 20344
I20260926 11:39:39.869266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:39:39.869566 392 external_mini_cluster.cc:1428] Setting key 16973f39fabd5278162e271c43311850
I20260926 11:39:39.871928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44005
--builtin_ntp_servers=127.0.98.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:39:39.874729 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.865839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47957 (local address 127.0.98.62:44005)
0926 11:39:39.865994 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:39.865998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:39.866987 (+ 989us) server_negotiation.cc:407] Connection header received
0926 11:39:39.868767 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:39.868770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:39.868839 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:39.868948 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:39.870250 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.870918 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.872382 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.872825 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.872962 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.874179 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:39.874190 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:39.874193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:39.874207 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:39.874229 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:39.874237 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:39.874327 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:39.874503 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 11:39:39.874578 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:39:39.875742 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:39.866082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44005 (local address 127.0.98.2:47957)
0926 11:39:39.866805 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:39.866851 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:39.868468 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:39.869181 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:39.869189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:39.869360 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:39.870021 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:39.870042 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.871072 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.871075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:39.872244 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:39.872257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:39.873008 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:39.873011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:39.873193 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:39.873881 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:39:39.873883 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:39.873929 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:39:39.874003 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:39.874329 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:39.874341 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:39.874349 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 11:39:39.874493 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 11:39:39.874839 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260926 11:39:39.877008 20472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44005
I20260926 11:39:39.877251 20472 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:39.877841 20472 heartbeater.cc:511] Master 127.0.98.62:44005 requested a full tablet report, sending...
I20260926 11:39:39.878959 20153 ts_manager.cc:194] Registered new tserver with Master: 95a24b41487d4ab0aadaf47b46bdad7f (127.0.98.2:42009)
I20260926 11:39:39.879560 20153 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47957
W20260926 11:39:39.987563 20475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:39.987982 20475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:39.988062 20475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:39.992192 20475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:39.992311 20475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:39.992400 20475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:39:39.997260 20475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:39.998512 20475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:40.000005 20475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:40.010840 20480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:40.011001 20481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:40.011228 20483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:40.011299 20475 server_base.cc:1034] running on GCE node
I20260926 11:39:40.011955 20475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:40.012719 20475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:40.013917 20475 hybrid_clock.cc:648] HybridClock initialized: now 1790422780013896 us; error 37 us; skew 500 ppm
I20260926 11:39:40.016618 20475 webserver.cc:460] Webserver started at http://127.0.98.3:37075/ using document root <none> and password file <none>
I20260926 11:39:40.017350 20475 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:40.017421 20475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:40.017717 20475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:40.020325 20475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/data/instance:
uuid: "df2860c207c5471fa31ebdb2f8f7b2ec"
format_stamp: "Formatted at 2026-09-26 11:39:40 on dist-test-slave-4pw5"
server_key: "d5369f40f47d7c713fb1c76eb8ed485e"
server_key_iv: "ed7a941e27eb11bfca03d5b6f5cc4b5e"
server_key_version: "encryptionkey@0"
I20260926 11:39:40.020979 20475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df2860c207c5471fa31ebdb2f8f7b2ec"
format_stamp: "Formatted at 2026-09-26 11:39:40 on dist-test-slave-4pw5"
server_key: "d5369f40f47d7c713fb1c76eb8ed485e"
server_key_iv: "ed7a941e27eb11bfca03d5b6f5cc4b5e"
server_key_version: "encryptionkey@0"
I20260926 11:39:40.026309 20475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:39:40.029196 20489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:40.030546 20475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:39:40.030726 20475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/wal
uuid: "df2860c207c5471fa31ebdb2f8f7b2ec"
format_stamp: "Formatted at 2026-09-26 11:39:40 on dist-test-slave-4pw5"
server_key: "d5369f40f47d7c713fb1c76eb8ed485e"
server_key_iv: "ed7a941e27eb11bfca03d5b6f5cc4b5e"
server_key_version: "encryptionkey@0"
I20260926 11:39:40.030865 20475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:40.062453 20475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:40.063290 20475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:40.063531 20475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:40.064250 20475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:39:40.065300 20475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:39:40.065378 20475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:40.065454 20475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:39:40.065497 20475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:40.075085 20475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46369
I20260926 11:39:40.075125 20602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46369 every 8 connection(s)
I20260926 11:39:40.076259 20475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:39:40.079775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 20475
I20260926 11:39:40.080075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:39:40.080526 392 external_mini_cluster.cc:1428] Setting key ff1cb56ade57565b159bed4492c76274
I20260926 11:39:40.089960 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.079927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34645 (local address 127.0.98.62:44005)
0926 11:39:40.080101 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:40.080105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:40.082053 (+ 1948us) server_negotiation.cc:407] Connection header received
0926 11:39:40.084128 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:40.084132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:40.084203 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:40.084304 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:40.085463 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.086112 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.087543 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.087847 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.087998 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.089404 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:40.089408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:40.089410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:40.089427 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:40.089449 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:40.089457 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:40.089527 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:40.089754 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 11:39:40.089825 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:39:40.090963 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.080747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44005 (local address 127.0.98.3:34645)
0926 11:39:40.081842 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:40.081918 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:40.083869 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:40.084523 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:40.084531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:40.084635 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:39:40.085285 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.085303 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.086287 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.086292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.087379 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.087391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.088050 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.088055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.088270 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.089077 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:39:40.089079 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:39:40.089126 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:39:40.089220 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:39:40.089553 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:39:40.089560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:39:40.089565 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:39:40.089698 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 11:39:40.089961 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":177,"threads_started":1}
I20260926 11:39:40.092522 20603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44005
I20260926 11:39:40.092833 20603 heartbeater.cc:464] Registering TS with master...
I20260926 11:39:40.093483 20603 heartbeater.cc:511] Master 127.0.98.62:44005 requested a full tablet report, sending...
I20260926 11:39:40.094951 20153 ts_manager.cc:194] Registered new tserver with Master: df2860c207c5471fa31ebdb2f8f7b2ec (127.0.98.3:46369)
I20260926 11:39:40.095767 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:39:40.095778 20153 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34645
I20260926 11:39:40.102823 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.098057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.0.98.62:44005)
0926 11:39:40.098195 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:40.098201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:40.098492 (+ 291us) server_negotiation.cc:407] Connection header received
0926 11:39:40.098561 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:40.098563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:40.098605 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:40.098682 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:40.099002 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.099523 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.100565 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.100790 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.100878 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.101983 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:40.101991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:40.101995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:40.102089 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:40.102123 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:40.102142 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:40.102251 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:40.102460 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 11:39:40.102551 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:39:40.106560 20153 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:39:40.109819 20153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:39:40.128361 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.120985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.0.98.3:46369)
0926 11:39:40.121406 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:40.121421 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:40.121772 (+ 351us) server_negotiation.cc:407] Connection header received
0926 11:39:40.121898 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:40.121905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:40.122174 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:40.122361 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:40.122755 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.124193 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.126274 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.127398 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.127531 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.127756 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:40.128002 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 11:39:40.128130 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":188,"threads_started":1}
I20260926 11:39:40.128343 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.120145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46369 (local address 127.0.0.1:34320)
0926 11:39:40.121640 (+ 1495us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:40.121655 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:40.121773 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:40.122367 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:40.122374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:40.122405 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:40.122601 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.122610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.124436 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.124439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.126046 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.126059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.127545 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.127549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.127768 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.127822 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 11:39:40.127969 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 11:39:40.128130 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"mutex_wait_us":131,"thread_start_us":78,"threads_started":1}
I20260926 11:39:40.128397 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.121068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.0.98.2:42009)
0926 11:39:40.121865 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:40.121878 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:40.122073 (+ 195us) server_negotiation.cc:407] Connection header received
0926 11:39:40.122221 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:40.122229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:40.122430 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:40.122590 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:40.123318 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.124637 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.126338 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.127206 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.127331 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.127541 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:40.128120 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 11:39:40.128249 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":135,"threads_started":1}
I20260926 11:39:40.128342 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.120135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42823 (local address 127.0.0.1:59386)
0926 11:39:40.121339 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:40.121405 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:40.121551 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:40.122564 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:40.122567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:40.122588 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:40.122707 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.122718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.124680 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.124685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.126178 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.126190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.127639 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.127643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.127856 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.127912 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 11:39:40.128001 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 11:39:40.128130 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":153,"thread_start_us":147,"threads_started":1}
I20260926 11:39:40.128803 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.120903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42009 (local address 127.0.0.1:56088)
0926 11:39:40.121963 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:40.121978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:40.122084 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:40.122762 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:40.122768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:40.123006 (+ 238us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:40.123160 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.123168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.124792 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.124795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.126181 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.126190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.127639 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.127643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.127876 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.127912 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 11:39:40.127990 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 11:39:40.128593 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":663,"thread_start_us":122,"threads_started":1}
I20260926 11:39:40.129065 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.121575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.0.98.1:42823)
0926 11:39:40.122005 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:40.122014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:40.122040 (+ 26us) server_negotiation.cc:407] Connection header received
0926 11:39:40.122131 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:40.122138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:40.122369 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:40.122602 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:40.122822 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.124563 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.126338 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.127206 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.127323 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.127541 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:40.128413 (+ 872us) server_negotiation.cc:299] Negotiation successful
0926 11:39:40.128909 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":121,"threads_started":1}
I20260926 11:39:40.132309 20406 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2f8263454349e3be11b5a9d6a72bb0 (DEFAULT_TABLE table=test-table [id=eacd204b7c0049a89dbc5234cb23613f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:40.133612 20406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d2f8263454349e3be11b5a9d6a72bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:40.133822 20275 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2f8263454349e3be11b5a9d6a72bb0 (DEFAULT_TABLE table=test-table [id=eacd204b7c0049a89dbc5234cb23613f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:40.134972 20275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d2f8263454349e3be11b5a9d6a72bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:40.140369 20537 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2f8263454349e3be11b5a9d6a72bb0 (DEFAULT_TABLE table=test-table [id=eacd204b7c0049a89dbc5234cb23613f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:39:40.141625 20537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d2f8263454349e3be11b5a9d6a72bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:40.142707 20620 tablet_bootstrap.cc:492] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: Bootstrap starting.
I20260926 11:39:40.144361 20621 tablet_bootstrap.cc:492] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765: Bootstrap starting.
I20260926 11:39:40.145442 20620 tablet_bootstrap.cc:654] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:40.146250 20620 log.cc:824] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:40.146826 20621 tablet_bootstrap.cc:654] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:40.147820 20621 log.cc:824] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:40.148444 20620 tablet_bootstrap.cc:492] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: No bootstrap required, opened a new log
I20260926 11:39:40.148669 20620 ts_tablet_manager.cc:1397] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:39:40.149945 20624 tablet_bootstrap.cc:492] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: Bootstrap starting.
I20260926 11:39:40.150558 20621 tablet_bootstrap.cc:492] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765: No bootstrap required, opened a new log
I20260926 11:39:40.150816 20621 ts_tablet_manager.cc:1397] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 11:39:40.152652 20624 tablet_bootstrap.cc:654] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:40.153609 20624 log.cc:824] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:40.153743 20620 raft_consensus.cc:348] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } }
I20260926 11:39:40.154055 20620 raft_consensus.cc:374] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:40.154134 20620 raft_consensus.cc:729] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a24b41487d4ab0aadaf47b46bdad7f, State: Initialized, Role: FOLLOWER
I20260926 11:39:40.154641 20620 consensus_queue.cc:260] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } }
I20260926 11:39:40.155100 20621 raft_consensus.cc:348] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } }
I20260926 11:39:40.155372 20472 heartbeater.cc:503] Master 127.0.98.62:44005 was elected leader, sending a full tablet report...
I20260926 11:39:40.155388 20621 raft_consensus.cc:374] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:40.155452 20621 raft_consensus.cc:729] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259ebb9958cf486b8177dc66a79ae765, State: Initialized, Role: FOLLOWER
I20260926 11:39:40.155774 20620 ts_tablet_manager.cc:1428] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 11:39:40.156031 20621 consensus_queue.cc:260] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } }
I20260926 11:39:40.156190 20624 tablet_bootstrap.cc:492] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: No bootstrap required, opened a new log
I20260926 11:39:40.156409 20624 ts_tablet_manager.cc:1397] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 11:39:40.156847 20341 heartbeater.cc:503] Master 127.0.98.62:44005 was elected leader, sending a full tablet report...
I20260926 11:39:40.158417 20621 ts_tablet_manager.cc:1428] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 11:39:40.161819 20624 raft_consensus.cc:348] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } }
I20260926 11:39:40.162106 20624 raft_consensus.cc:374] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:40.162211 20624 raft_consensus.cc:729] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2860c207c5471fa31ebdb2f8f7b2ec, State: Initialized, Role: FOLLOWER
I20260926 11:39:40.162786 20624 consensus_queue.cc:260] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } }
I20260926 11:39:40.163543 20603 heartbeater.cc:503] Master 127.0.98.62:44005 was elected leader, sending a full tablet report...
I20260926 11:39:40.163924 20624 ts_tablet_manager.cc:1428] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260926 11:39:40.327826 20604 tablet.cc:2369] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:39:40.365041 20473 tablet.cc:2369] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:39:40.407147 20342 tablet.cc:2369] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:39:40.410543 20626 raft_consensus.cc:482] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:39:40.410799 20626 raft_consensus.cc:504] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } }
I20260926 11:39:40.412250 20626 leader_election.cc:288] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2860c207c5471fa31ebdb2f8f7b2ec (127.0.98.3:46369), 259ebb9958cf486b8177dc66a79ae765 (127.0.98.1:42823)
I20260926 11:39:40.416352 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.412692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56273 (local address 127.0.98.3:46369)
0926 11:39:40.412824 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:40.412838 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:40.412969 (+ 131us) server_negotiation.cc:407] Connection header received
0926 11:39:40.413201 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:40.413212 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:40.413269 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:40.413348 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:40.413610 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.414128 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.415294 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.415783 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.415921 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.415987 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:40.416145 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 11:39:40.416224 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 11:39:40.416369 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.412473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46369 (local address 127.0.98.2:56273)
0926 11:39:40.412862 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:40.412877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:40.413022 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:40.413373 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:40.413376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:40.413391 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:40.413505 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.413511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.414251 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.414257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.415169 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.415179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.415916 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.415919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.416008 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.416039 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 11:39:40.416129 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 11:39:40.416209 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":95,"threads_started":1}
I20260926 11:39:40.417065 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.412478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42823 (local address 127.0.98.2:33969)
0926 11:39:40.413024 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:40.413045 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 11:39:40.413209 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:39:40.413506 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:39:40.413510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:39:40.413527 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:39:40.413672 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.413679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.414450 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.414453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.415387 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:39:40.415396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.416336 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.416340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:39:40.416678 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.416703 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 11:39:40.416810 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 11:39:40.416893 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":89,"thread_start_us":67,"threads_started":1}
I20260926 11:39:40.417052 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.412647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33969 (local address 127.0.98.1:42823)
0926 11:39:40.412831 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:40.412837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:40.413135 (+ 298us) server_negotiation.cc:407] Connection header received
0926 11:39:40.413339 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:40.413341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:40.413388 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:40.413483 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:39:40.413796 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.414335 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.415523 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.416197 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.416322 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.416384 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:40.416859 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 11:39:40.416941 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 11:39:40.417117 20557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2f8263454349e3be11b5a9d6a72bb0" candidate_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" is_pre_election: true
I20260926 11:39:40.417475 20557 raft_consensus.cc:2454] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95a24b41487d4ab0aadaf47b46bdad7f in term 0.
I20260926 11:39:40.417866 20295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2f8263454349e3be11b5a9d6a72bb0" candidate_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259ebb9958cf486b8177dc66a79ae765" is_pre_election: true
I20260926 11:39:40.418129 20360 leader_election.cc:302] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a24b41487d4ab0aadaf47b46bdad7f, df2860c207c5471fa31ebdb2f8f7b2ec; no voters:
I20260926 11:39:40.418459 20626 raft_consensus.cc:2781] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:39:40.418565 20626 raft_consensus.cc:482] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:39:40.418521 20295 raft_consensus.cc:2454] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95a24b41487d4ab0aadaf47b46bdad7f in term 0.
I20260926 11:39:40.418627 20626 raft_consensus.cc:3037] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:40.419843 20626 raft_consensus.cc:504] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } }
I20260926 11:39:40.420256 20626 leader_election.cc:288] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [CANDIDATE]: Term 1 election: Requested vote from peers df2860c207c5471fa31ebdb2f8f7b2ec (127.0.98.3:46369), 259ebb9958cf486b8177dc66a79ae765 (127.0.98.1:42823)
I20260926 11:39:40.420641 20295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2f8263454349e3be11b5a9d6a72bb0" candidate_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259ebb9958cf486b8177dc66a79ae765"
I20260926 11:39:40.420776 20557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2f8263454349e3be11b5a9d6a72bb0" candidate_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec"
I20260926 11:39:40.420782 20295 raft_consensus.cc:3037] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:40.420934 20557 raft_consensus.cc:3037] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:40.421998 20557 raft_consensus.cc:2454] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a24b41487d4ab0aadaf47b46bdad7f in term 1.
I20260926 11:39:40.421999 20295 raft_consensus.cc:2454] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a24b41487d4ab0aadaf47b46bdad7f in term 1.
I20260926 11:39:40.422382 20360 leader_election.cc:302] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a24b41487d4ab0aadaf47b46bdad7f, df2860c207c5471fa31ebdb2f8f7b2ec; no voters:
I20260926 11:39:40.422658 20626 raft_consensus.cc:2781] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:40.422933 20626 raft_consensus.cc:686] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 1 LEADER]: Becoming Leader. State: Replica: 95a24b41487d4ab0aadaf47b46bdad7f, State: Running, Role: LEADER
I20260926 11:39:40.423296 20626 consensus_queue.cc:237] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } }
I20260926 11:39:40.426877 20153 catalog_manager.cc:5495] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 95a24b41487d4ab0aadaf47b46bdad7f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a24b41487d4ab0aadaf47b46bdad7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42009 } health_report { overall_health: HEALTHY } } }
I20260926 11:39:40.493002 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.489309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.0.98.2:42009)
0926 11:39:40.489460 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:40.489464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:40.489477 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:39:40.489590 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:40.489592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:40.489649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:40.489731 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:39:40.489956 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.490451 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.491430 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:40.491756 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:40.491848 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:40.492135 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:39:40.492591 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:39:40.492691 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:40.492831 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 11:39:40.492886 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 11:39:40.499266 20557 raft_consensus.cc:1261] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [term 1 FOLLOWER]: Refusing update from remote peer 95a24b41487d4ab0aadaf47b46bdad7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:40.499266 20295 raft_consensus.cc:1261] T 4d2f8263454349e3be11b5a9d6a72bb0 P 259ebb9958cf486b8177dc66a79ae765 [term 1 FOLLOWER]: Refusing update from remote peer 95a24b41487d4ab0aadaf47b46bdad7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:39:40.500063 20631 consensus_queue.cc:1035] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2860c207c5471fa31ebdb2f8f7b2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:40.500291 20626 consensus_queue.cc:1035] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "259ebb9958cf486b8177dc66a79ae765" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:39:40.507822 20634 mvcc.cc:205] Tried to move back new op lower bound from 7333571708917149696 to 7333571708616814592. Current Snapshot: MvccSnapshot[applied={T|T < 7333571708917149696}]
I20260926 11:39:40.509262 20635 mvcc.cc:205] Tried to move back new op lower bound from 7333571708917149696 to 7333571708616814592. Current Snapshot: MvccSnapshot[applied={T|T < 7333571708917149696}]
I20260926 11:39:40.512058 20636 mvcc.cc:205] Tried to move back new op lower bound from 7333571708917149696 to 7333571708616814592. Current Snapshot: MvccSnapshot[applied={T|T < 7333571708917149696}]
I20260926 11:39:40.516762 20152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:40.516973 20152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:39:40.519690 20152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 649e722ad088491b8eab3e0d43f922b3: Sending DeleteTablet for 3 replicas of tablet 4d2f8263454349e3be11b5a9d6a72bb0
I20260926 11:39:40.520581 20537 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d2f8263454349e3be11b5a9d6a72bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:40 UTC) from {username='slave'} at 127.0.0.1:34320
I20260926 11:39:40.520599 20406 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d2f8263454349e3be11b5a9d6a72bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:40 UTC) from {username='slave'} at 127.0.0.1:56088
I20260926 11:39:40.521162 20644 tablet_replica.cc:330] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: stopping tablet replica
I20260926 11:39:40.521390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 20213
I20260926 11:39:40.521363 20275 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d2f8263454349e3be11b5a9d6a72bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:39:40 UTC) from {username='slave'} at 127.0.0.1:59386
I20260926 11:39:40.521693 20644 raft_consensus.cc:2229] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:39:40.522136 20644 raft_consensus.cc:2258] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:40.522717 20645 tablet_replica.cc:330] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: stopping tablet replica
I20260926 11:39:40.523087 20645 raft_consensus.cc:2229] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:39:40.523471 20645 raft_consensus.cc:2258] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:39:40.523777 20644 ts_tablet_manager.cc:1905] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:40.525149 20645 ts_tablet_manager.cc:1905] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:39:40.526906 20644 ts_tablet_manager.cc:1918] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:40.527021 20644 log.cc:1196] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-1/wal/wals/4d2f8263454349e3be11b5a9d6a72bb0
I20260926 11:39:40.527372 20644 ts_tablet_manager.cc:1939] T 4d2f8263454349e3be11b5a9d6a72bb0 P 95a24b41487d4ab0aadaf47b46bdad7f: Deleting consensus metadata
I20260926 11:39:40.528438 20645 ts_tablet_manager.cc:1918] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:39:40.528607 20645 log.cc:1196] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: Deleting WAL directory at /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422511662684-392-0/minicluster-data/ts-2/wal/wals/4d2f8263454349e3be11b5a9d6a72bb0
I20260926 11:39:40.529003 20645 ts_tablet_manager.cc:1939] T 4d2f8263454349e3be11b5a9d6a72bb0 P df2860c207c5471fa31ebdb2f8f7b2ec: Deleting consensus metadata
I20260926 11:39:40.529075 20139 catalog_manager.cc:4841] TS 95a24b41487d4ab0aadaf47b46bdad7f (127.0.98.2:42009): tablet 4d2f8263454349e3be11b5a9d6a72bb0 (table test-table [id=eacd204b7c0049a89dbc5234cb23613f]) successfully deleted
I20260926 11:39:40.530344 20138 catalog_manager.cc:4841] TS df2860c207c5471fa31ebdb2f8f7b2ec (127.0.98.3:46369): tablet 4d2f8263454349e3be11b5a9d6a72bb0 (table test-table [id=eacd204b7c0049a89dbc5234cb23613f]) successfully deleted
W20260926 11:39:40.530974 20139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42823 (error 108)
I20260926 11:39:40.531200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 20344
I20260926 11:39:40.532258 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:40.531957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42823 (local address 127.0.0.1:59388)
0926 11:39:40.532098 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 11:39:40.532178 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 11:39:40.532456 20139 catalog_manager.cc:4584] TS 259ebb9958cf486b8177dc66a79ae765 (127.0.98.1:42823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d2f8263454349e3be11b5a9d6a72bb0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42823: connect: Connection refused (error 111)
I20260926 11:39:40.537199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 20475
I20260926 11:39:40.542856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 20122
2026-09-26T11:39:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1386 ms)
[----------] 6 tests from EncryptionPolicyTest (11170 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:40 dist-test-slave-4pw5 krb5kdc[20649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:40 dist-test-slave-4pw5 krb5kdc[20649](info): set up 2 sockets
Sep 26 11:39:40 dist-test-slave-4pw5 krb5kdc[20649](info): commencing operation
krb5kdc: starting...
W20260926 11:39:41.628862 392 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:41 dist-test-slave-4pw5 krb5kdc[20649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:41.776726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:41.911724 20662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:41.912199 20662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:41.912288 20662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:41.916847 20662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:41.916988 20662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:41.917030 20662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:41.917071 20662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:41.917109 20662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:41.923466 20662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:41.925091 20662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:41.926465 20662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:41.937584 20668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:41.937606 20670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:41.937610 20667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:41.938117 20662 server_base.cc:1034] running on GCE node
I20260926 11:39:41.938879 20662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:41.940506 20662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:41.941709 20662 hybrid_clock.cc:648] HybridClock initialized: now 1790422781941691 us; error 42 us; skew 500 ppm
Sep 26 11:39:41 dist-test-slave-4pw5 krb5kdc[20649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:41.945924 20662 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:41.947304 20662 webserver.cc:460] Webserver started at http://127.0.0.1:46797/ using document root <none> and password file <none>
I20260926 11:39:41.948159 20662 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:41.948271 20662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:41.948578 20662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:41.951380 20662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "118d6f5a42454bca807c9e65feda5819"
format_stamp: "Formatted at 2026-09-26 11:39:41 on dist-test-slave-4pw5"
server_key: "fd7a67808d58ef2118cced6bebc266b4"
server_key_iv: "af6a7641441ca443053da3803f8c8cbd"
server_key_version: "encryptionkey@0"
I20260926 11:39:41.952283 20662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "118d6f5a42454bca807c9e65feda5819"
format_stamp: "Formatted at 2026-09-26 11:39:41 on dist-test-slave-4pw5"
server_key: "fd7a67808d58ef2118cced6bebc266b4"
server_key_iv: "af6a7641441ca443053da3803f8c8cbd"
server_key_version: "encryptionkey@0"
I20260926 11:39:41.957571 20662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 11:39:41.961345 20677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:41.963162 20662 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 11:39:41.963380 20662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "118d6f5a42454bca807c9e65feda5819"
format_stamp: "Formatted at 2026-09-26 11:39:41 on dist-test-slave-4pw5"
server_key: "fd7a67808d58ef2118cced6bebc266b4"
server_key_iv: "af6a7641441ca443053da3803f8c8cbd"
server_key_version: "encryptionkey@0"
I20260926 11:39:41.963577 20662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:41.982993 20662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:41.987602 20662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:41.987888 20662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:41.997288 20662 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38839
I20260926 11:39:41.997323 20729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38839 every 8 connection(s)
I20260926 11:39:41.998510 20662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:42.001758 20730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:42.006291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 20662
I20260926 11:39:42.006453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:42.006798 392 external_mini_cluster.cc:1428] Setting key d7504daaa772c50b32e6c741c1e84c9e
I20260926 11:39:42.009253 20730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819: Bootstrap starting.
I20260926 11:39:42.011808 20730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:42.012817 20730 log.cc:824] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:42.015072 20730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819: No bootstrap required, opened a new log
Sep 26 11:39:42 dist-test-slave-4pw5 krb5kdc[20649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 11:39:42.019265 20730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118d6f5a42454bca807c9e65feda5819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } }
I20260926 11:39:42.019646 20730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:42.019735 20730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118d6f5a42454bca807c9e65feda5819, State: Initialized, Role: FOLLOWER
I20260926 11:39:42.020416 20730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118d6f5a42454bca807c9e65feda5819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } }
I20260926 11:39:42.020630 20730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:42.020713 20730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:42.020828 20730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:42.021860 20730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118d6f5a42454bca807c9e65feda5819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } }
I20260926 11:39:42.022284 20730 leader_election.cc:302] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 118d6f5a42454bca807c9e65feda5819; no voters:
I20260926 11:39:42.022635 20730 leader_election.cc:288] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:42.022816 20735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:42.023115 20735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [term 1 LEADER]: Becoming Leader. State: Replica: 118d6f5a42454bca807c9e65feda5819, State: Running, Role: LEADER
I20260926 11:39:42.023522 20735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118d6f5a42454bca807c9e65feda5819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } }
I20260926 11:39:42.024322 20730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:42.025396 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:42.008556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48611 (local address 127.0.0.1:38839)
0926 11:39:42.009094 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:42.009111 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:42.009165 (+ 54us) server_negotiation.cc:407] Connection header received
0926 11:39:42.009964 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:42.009984 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:42.010372 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:42.010756 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:42.011668 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:42.013127 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:42.014245 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:42.014802 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:42.014993 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:42.018162 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:42.018174 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:42.018190 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:42.018223 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:42.020645 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:42.021387 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:42.021393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:42.021398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:42.021539 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:42.021979 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:42.022005 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:42.022010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:42.022582 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:42.022864 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:42.023924 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0926 11:39:42.024210 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":175,"threads_started":1}
I20260926 11:39:42.025547 20736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "118d6f5a42454bca807c9e65feda5819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118d6f5a42454bca807c9e65feda5819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } } }
I20260926 11:39:42.025688 20736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:42.025924 20737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118d6f5a42454bca807c9e65feda5819. Latest consensus state: current_term: 1 leader_uuid: "118d6f5a42454bca807c9e65feda5819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118d6f5a42454bca807c9e65feda5819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } } }
I20260926 11:39:42.026058 20737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:42.026247 20744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:42.029690 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:42.030277 20744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:42.037808 20744 catalog_manager.cc:1347] Generated new cluster ID: e962d2760c4e401ea1453d8676e47274
I20260926 11:39:42.037900 20744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:42.045001 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:42.032718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40231 (local address 127.0.0.1:38839)
0926 11:39:42.032863 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:42.032870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:42.034080 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 11:39:42.034286 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:42.034290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:42.034346 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:42.034400 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:42.035395 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:42.036862 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:42.038118 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:42.038479 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:42.038594 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:42.040586 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:42.040591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:42.040594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:42.040631 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:42.042698 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:42.043323 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:42.043327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:42.043329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:42.043427 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:42.043888 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:42.043892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:42.043893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:42.044140 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:42.044320 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:42.044678 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 11:39:42.044794 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 11:39:42.064913 20744 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:42.066107 20744 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:42.081387 20744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 118d6f5a42454bca807c9e65feda5819: Generated new TSK 0
I20260926 11:39:42.082571 20744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:42.099006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 20662
W20260926 11:39:42.104302 20647 connection.cc:535] client connection to 127.0.0.1:38839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38839): Connection reset by peer (error 104)
2026-09-26T11:39:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1553 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:42 dist-test-slave-4pw5 krb5kdc[20761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:42 dist-test-slave-4pw5 krb5kdc[20761](info): set up 2 sockets
Sep 26 11:39:42 dist-test-slave-4pw5 krb5kdc[20761](info): commencing operation
krb5kdc: starting...
W20260926 11:39:43.244479 392 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:43 dist-test-slave-4pw5 krb5kdc[20761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:43.380505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:43.501816 20774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:43.502163 20774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:43.502261 20774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:43.506434 20774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:43.506542 20774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:43.506582 20774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:43.506639 20774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:43.506676 20774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:43.512372 20774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34961
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:43.513657 20774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:43.514802 20774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:43.523838 20780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:43.523838 20779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:43.524072 20774 server_base.cc:1034] running on GCE node
W20260926 11:39:43.523856 20782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:43.524904 20774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:43.526183 20774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:43.527390 20774 hybrid_clock.cc:648] HybridClock initialized: now 1790422783527369 us; error 39 us; skew 500 ppm
Sep 26 11:39:43 dist-test-slave-4pw5 krb5kdc[20761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:43.530303 20774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:43.531226 20774 webserver.cc:460] Webserver started at http://127.0.0.1:45593/ using document root <none> and password file <none>
I20260926 11:39:43.531872 20774 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:43.531940 20774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:43.532125 20774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:43.534484 20774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "6cf5e07d95df449ba5989adc7ba49f57"
format_stamp: "Formatted at 2026-09-26 11:39:43 on dist-test-slave-4pw5"
server_key: "8b4b653faaa62a1da08b6ab9dda49a85"
server_key_iv: "fa12ad0d241d2f17fb206fa017215684"
server_key_version: "encryptionkey@0"
I20260926 11:39:43.535034 20774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "6cf5e07d95df449ba5989adc7ba49f57"
format_stamp: "Formatted at 2026-09-26 11:39:43 on dist-test-slave-4pw5"
server_key: "8b4b653faaa62a1da08b6ab9dda49a85"
server_key_iv: "fa12ad0d241d2f17fb206fa017215684"
server_key_version: "encryptionkey@0"
I20260926 11:39:43.539147 20774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:39:43.541785 20789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:43.543332 20774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 11:39:43.543450 20774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "6cf5e07d95df449ba5989adc7ba49f57"
format_stamp: "Formatted at 2026-09-26 11:39:43 on dist-test-slave-4pw5"
server_key: "8b4b653faaa62a1da08b6ab9dda49a85"
server_key_iv: "fa12ad0d241d2f17fb206fa017215684"
server_key_version: "encryptionkey@0"
I20260926 11:39:43.543556 20774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:43.583079 20774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:43.586707 20774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:43.586884 20774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:43.595088 20774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34961
I20260926 11:39:43.595103 20841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34961 every 8 connection(s)
I20260926 11:39:43.596375 20774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:43.598519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 20774
I20260926 11:39:43.598706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:43.599030 392 external_mini_cluster.cc:1428] Setting key a1614f15808c00378aa14093f78eb0af
I20260926 11:39:43.599742 20842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:43.606321 20842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57: Bootstrap starting.
I20260926 11:39:43.609027 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:43.600674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36561 (local address 127.0.0.1:34961)
0926 11:39:43.601211 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:43.601230 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:43.601282 (+ 52us) server_negotiation.cc:407] Connection header received
0926 11:39:43.602007 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:43.602027 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:43.602411 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:43.602803 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:43.603244 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:43.604352 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:43.605322 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:43.605850 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:43.606044 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:43.607144 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:43.607150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:43.607164 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:43.607645 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:43.607719 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:43.607746 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:43.607904 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:43.608112 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 11:39:43.608268 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":148,"threads_started":1}
I20260926 11:39:43.609009 20842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:43.610140 20842 log.cc:824] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:43.612777 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:43.612970 20842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57: No bootstrap required, opened a new log
I20260926 11:39:43.617563 20842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf5e07d95df449ba5989adc7ba49f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260926 11:39:43.617904 20842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:43.618000 20842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf5e07d95df449ba5989adc7ba49f57, State: Initialized, Role: FOLLOWER
I20260926 11:39:43.618515 20842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf5e07d95df449ba5989adc7ba49f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260926 11:39:43.618693 20842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:43.618798 20842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:43.618963 20842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:43.620141 20842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf5e07d95df449ba5989adc7ba49f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260926 11:39:43.620455 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:43.614993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51053 (local address 127.0.0.1:34961)
0926 11:39:43.615123 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:43.615128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:43.615324 (+ 196us) server_negotiation.cc:407] Connection header received
0926 11:39:43.615559 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:43.615562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:43.615646 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:43.615748 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:43.616035 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:43.617065 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:43.618201 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:43.618455 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:43.618579 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:43.619670 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:43.619674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:43.619677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:43.619692 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:43.619725 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:43.619737 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:43.619844 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:43.619975 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 11:39:43.620021 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:39:43.620739 20842 leader_election.cc:302] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cf5e07d95df449ba5989adc7ba49f57; no voters:
I20260926 11:39:43.621145 20842 leader_election.cc:288] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:43.621234 20852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:43.621438 20852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf5e07d95df449ba5989adc7ba49f57, State: Running, Role: LEADER
I20260926 11:39:43.621809 20852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf5e07d95df449ba5989adc7ba49f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260926 11:39:43.622587 20842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:43.623731 20854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cf5e07d95df449ba5989adc7ba49f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf5e07d95df449ba5989adc7ba49f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } } }
I20260926 11:39:43.623768 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf5e07d95df449ba5989adc7ba49f57. Latest consensus state: current_term: 1 leader_uuid: "6cf5e07d95df449ba5989adc7ba49f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf5e07d95df449ba5989adc7ba49f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } } }
I20260926 11:39:43.623886 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:43.623886 20854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:43.624437 20861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:43.627960 20861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:43.634944 20861 catalog_manager.cc:1347] Generated new cluster ID: b16ac3e86ddc4a09b53c74b46f3643f4
I20260926 11:39:43.635046 20861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:43.655895 20861 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:43.657544 20861 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:43.670612 20861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cf5e07d95df449ba5989adc7ba49f57: Generated new TSK 0
I20260926 11:39:43.671684 20861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:43.677438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 20774
W20260926 11:39:43.683022 20759 connection.cc:535] client connection to 127.0.0.1:34961 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34961): Connection reset by peer (error 104)
2026-09-26T11:39:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1578 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:43 dist-test-slave-4pw5 krb5kdc[20873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:43 dist-test-slave-4pw5 krb5kdc[20873](info): set up 2 sockets
Sep 26 11:39:43 dist-test-slave-4pw5 krb5kdc[20873](info): commencing operation
krb5kdc: starting...
W20260926 11:39:44.743959 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:44 dist-test-slave-4pw5 krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:44.883374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32933
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:44.998551 20886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:44.998869 20886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:44.998914 20886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:45.003265 20886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:45.003362 20886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:45.003381 20886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:45.003399 20886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:45.003414 20886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:45.009958 20886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32933
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:45.011221 20886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:45.012377 20886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:45.021968 20892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:45.022033 20886 server_base.cc:1034] running on GCE node
W20260926 11:39:45.022002 20891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:45.021972 20894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:45.022969 20886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:45.024250 20886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:45.025477 20886 hybrid_clock.cc:648] HybridClock initialized: now 1790422785025446 us; error 46 us; skew 500 ppm
Sep 26 11:39:45 dist-test-slave-4pw5 krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:45.032380 20886 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:45.033393 20886 webserver.cc:460] Webserver started at http://127.0.0.1:33103/ using document root <none> and password file <none>
I20260926 11:39:45.034017 20886 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:45.034114 20886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:45.034363 20886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:45.036623 20886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "6215fdd7979f412fb22714b38927ff62"
format_stamp: "Formatted at 2026-09-26 11:39:45 on dist-test-slave-4pw5"
server_key: "163a2f9eb3c6f0f2668e97dd8cdf6621"
server_key_iv: "2627896c716ac39b2c53caae9d698486"
server_key_version: "encryptionkey@0"
I20260926 11:39:45.037222 20886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "6215fdd7979f412fb22714b38927ff62"
format_stamp: "Formatted at 2026-09-26 11:39:45 on dist-test-slave-4pw5"
server_key: "163a2f9eb3c6f0f2668e97dd8cdf6621"
server_key_iv: "2627896c716ac39b2c53caae9d698486"
server_key_version: "encryptionkey@0"
I20260926 11:39:45.041100 20886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:45.043900 20901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:45.046269 20886 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:39:45.046483 20886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "6215fdd7979f412fb22714b38927ff62"
format_stamp: "Formatted at 2026-09-26 11:39:45 on dist-test-slave-4pw5"
server_key: "163a2f9eb3c6f0f2668e97dd8cdf6621"
server_key_iv: "2627896c716ac39b2c53caae9d698486"
server_key_version: "encryptionkey@0"
I20260926 11:39:45.046658 20886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:45.051892 20886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:45.055403 20886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:45.055694 20886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:45.062780 20886 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32933
I20260926 11:39:45.062841 20953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32933 every 8 connection(s)
I20260926 11:39:45.063840 20886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:45.066841 20954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:45.070498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 20886
I20260926 11:39:45.070696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:45.071015 392 external_mini_cluster.cc:1428] Setting key 3c1005b499ecdad84ca4bdf7a6f54c0b
I20260926 11:39:45.073339 20954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62: Bootstrap starting.
I20260926 11:39:45.076229 20954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:45.076720 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:45.072479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41123 (local address 127.0.0.1:32933)
0926 11:39:45.072938 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:45.072953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:45.072992 (+ 39us) server_negotiation.cc:407] Connection header received
0926 11:39:45.073727 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:45.073749 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:45.074060 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:45.074377 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:45.075317 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:45.075321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:45.075333 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:45.075344 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:45.075400 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:45.075419 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:45.075530 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:45.075696 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 11:39:45.075879 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":166,"threads_started":1}
I20260926 11:39:45.077131 20954 log.cc:824] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:45.079337 20954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62: No bootstrap required, opened a new log
I20260926 11:39:45.079738 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:45.083657 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:45.081858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56573 (local address 127.0.0.1:32933)
0926 11:39:45.081982 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:45.081985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:45.082205 (+ 220us) server_negotiation.cc:407] Connection header received
0926 11:39:45.082288 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:45.082291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:45.082331 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:45.082424 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:45.083179 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:45.083183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:45.083186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:45.083198 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:45.083228 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:45.083241 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:45.083340 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:45.083461 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 11:39:45.083502 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 11:39:45.084028 20954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6215fdd7979f412fb22714b38927ff62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32933 } }
I20260926 11:39:45.084275 20954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:45.084311 20954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6215fdd7979f412fb22714b38927ff62, State: Initialized, Role: FOLLOWER
I20260926 11:39:45.084758 20954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6215fdd7979f412fb22714b38927ff62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32933 } }
I20260926 11:39:45.084913 20954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:45.084986 20954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:45.085129 20954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:45.086149 20954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6215fdd7979f412fb22714b38927ff62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32933 } }
I20260926 11:39:45.086530 20954 leader_election.cc:302] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6215fdd7979f412fb22714b38927ff62; no voters:
I20260926 11:39:45.086874 20954 leader_election.cc:288] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:45.087018 20964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:45.087332 20964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [term 1 LEADER]: Becoming Leader. State: Replica: 6215fdd7979f412fb22714b38927ff62, State: Running, Role: LEADER
I20260926 11:39:45.087742 20964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6215fdd7979f412fb22714b38927ff62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32933 } }
I20260926 11:39:45.088266 20954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:45.089550 20966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6215fdd7979f412fb22714b38927ff62. Latest consensus state: current_term: 1 leader_uuid: "6215fdd7979f412fb22714b38927ff62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6215fdd7979f412fb22714b38927ff62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32933 } } }
I20260926 11:39:45.089691 20966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:45.090068 20965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6215fdd7979f412fb22714b38927ff62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6215fdd7979f412fb22714b38927ff62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32933 } } }
I20260926 11:39:45.090157 20973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:45.090178 20965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:45.094806 20973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:45.101632 20973 catalog_manager.cc:1347] Generated new cluster ID: 351e319734ba48fd8b682e03d0a4b3d6
I20260926 11:39:45.101728 20973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:45.114601 20973 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:45.115028 20973 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:45.123019 20973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6215fdd7979f412fb22714b38927ff62: Generated new TSK 0
I20260926 11:39:45.123975 20973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:45.128582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 20886
2026-09-26T11:39:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:45 dist-test-slave-4pw5 krb5kdc[20985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:45 dist-test-slave-4pw5 krb5kdc[20985](info): set up 2 sockets
Sep 26 11:39:45 dist-test-slave-4pw5 krb5kdc[20985](info): commencing operation
krb5kdc: starting...
W20260926 11:39:46.201678 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:46 dist-test-slave-4pw5 krb5kdc[20985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:46.365571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:46.499879 20998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:46.500303 20998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:46.500396 20998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:46.505229 20998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:46.505343 20998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:46.505378 20998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:46.505421 20998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:46.505477 20998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:46.511763 20998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:46.513319 20998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:46.514659 20998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:39:46.525874 20998 server_base.cc:1034] running on GCE node
W20260926 11:39:46.525759 21004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:46.525759 21006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:46.525841 21003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:46.527010 20998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:46.528642 20998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:46.529860 20998 hybrid_clock.cc:648] HybridClock initialized: now 1790422786529826 us; error 58 us; skew 500 ppm
Sep 26 11:39:46 dist-test-slave-4pw5 krb5kdc[20985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:46.533569 20998 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:46.534865 20998 webserver.cc:460] Webserver started at http://127.0.0.1:42479/ using document root <none> and password file <none>
I20260926 11:39:46.535732 20998 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:46.535846 20998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:46.536235 20998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:46.539167 20998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "1220f3c602b441f391dc6361eeb845ac"
format_stamp: "Formatted at 2026-09-26 11:39:46 on dist-test-slave-4pw5"
server_key: "7d3892c713b5ac10e5bbe82b10511731"
server_key_iv: "4e7713bd0cd24bbae570c43e7c8296f6"
server_key_version: "encryptionkey@0"
I20260926 11:39:46.539980 20998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "1220f3c602b441f391dc6361eeb845ac"
format_stamp: "Formatted at 2026-09-26 11:39:46 on dist-test-slave-4pw5"
server_key: "7d3892c713b5ac10e5bbe82b10511731"
server_key_iv: "4e7713bd0cd24bbae570c43e7c8296f6"
server_key_version: "encryptionkey@0"
I20260926 11:39:46.546000 20998 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 11:39:46.550192 21013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:46.552363 20998 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:39:46.552671 20998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "1220f3c602b441f391dc6361eeb845ac"
format_stamp: "Formatted at 2026-09-26 11:39:46 on dist-test-slave-4pw5"
server_key: "7d3892c713b5ac10e5bbe82b10511731"
server_key_iv: "4e7713bd0cd24bbae570c43e7c8296f6"
server_key_version: "encryptionkey@0"
I20260926 11:39:46.552870 20998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:46.574556 20998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:46.579349 20998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:46.579690 20998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:46.589124 20998 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45413
I20260926 11:39:46.589135 21065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45413 every 8 connection(s)
I20260926 11:39:46.590480 20998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:46.594715 21066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:46.595682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 20998
I20260926 11:39:46.595850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:46.596246 392 external_mini_cluster.cc:1428] Setting key 5712b8ed399f863acf91c2013a7b3d1b
I20260926 11:39:46.603968 21066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac: Bootstrap starting.
I20260926 11:39:46.607324 21066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac: Neither blocks nor log segments found. Creating new log.
Sep 26 11:39:46 dist-test-slave-4pw5 krb5kdc[20985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 11:39:46.608625 21066 log.cc:824] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:46.611367 21066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac: No bootstrap required, opened a new log
I20260926 11:39:46.616024 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:46.598398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59857 (local address 127.0.0.1:45413)
0926 11:39:46.599079 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:46.599103 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:46.599154 (+ 51us) server_negotiation.cc:407] Connection header received
0926 11:39:46.600078 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:46.600104 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:46.600620 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:46.601060 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:46.602158 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:46.603675 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:46.604814 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:46.605415 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:46.605650 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:46.609354 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:46.609361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:46.609382 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:46.609416 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:46.612014 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:46.612713 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:46.612720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:46.612727 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:46.612884 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:46.613251 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:46.613254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:46.613255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:46.613806 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:46.614044 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:46.614400 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 11:39:46.614858 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":200,"threads_started":1}
I20260926 11:39:46.616677 21066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1220f3c602b441f391dc6361eeb845ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45413 } }
I20260926 11:39:46.617072 21066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:46.617132 21066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1220f3c602b441f391dc6361eeb845ac, State: Initialized, Role: FOLLOWER
I20260926 11:39:46.617712 21066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1220f3c602b441f391dc6361eeb845ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45413 } }
I20260926 11:39:46.617869 21066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:46.617954 21066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:46.618090 21066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:46.619620 21066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1220f3c602b441f391dc6361eeb845ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45413 } }
I20260926 11:39:46.620095 21066 leader_election.cc:302] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1220f3c602b441f391dc6361eeb845ac; no voters:
I20260926 11:39:46.620478 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:46.620548 21066 leader_election.cc:288] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:46.620689 21071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:46.620908 21071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [term 1 LEADER]: Becoming Leader. State: Replica: 1220f3c602b441f391dc6361eeb845ac, State: Running, Role: LEADER
I20260926 11:39:46.621296 21071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1220f3c602b441f391dc6361eeb845ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45413 } }
I20260926 11:39:46.622152 21066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:46.622876 21071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1220f3c602b441f391dc6361eeb845ac. Latest consensus state: current_term: 1 leader_uuid: "1220f3c602b441f391dc6361eeb845ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1220f3c602b441f391dc6361eeb845ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45413 } } }
I20260926 11:39:46.623031 21071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:46.623433 21072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1220f3c602b441f391dc6361eeb845ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1220f3c602b441f391dc6361eeb845ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45413 } } }
I20260926 11:39:46.623548 21072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:46.624583 21081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:46.629199 21081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:46.634084 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:46.623910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45563 (local address 127.0.0.1:45413)
0926 11:39:46.624069 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:46.624076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:46.624093 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:39:46.624144 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:46.624148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:46.624212 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:46.625330 (+ 1118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:46.625440 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:46.626231 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:46.627469 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:46.627862 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:46.627975 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:46.629982 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:46.629986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:46.629989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:46.630022 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:46.632057 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:46.632623 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:46.632627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:46.632629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:46.632733 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:46.633036 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:46.633040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:46.633042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:46.633285 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:46.633441 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:46.633691 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 11:39:46.633842 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:39:46.638092 21081 catalog_manager.cc:1347] Generated new cluster ID: 9c03c2fa634a4a9bad47b393762ab0b2
I20260926 11:39:46.638211 21081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:46.660836 21081 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:46.662693 21081 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:46.669726 21081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1220f3c602b441f391dc6361eeb845ac: Generated new TSK 0
I20260926 11:39:46.670801 21081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:46.683291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 20998
2026-09-26T11:39:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1557 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:46 dist-test-slave-4pw5 krb5kdc[21097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:46 dist-test-slave-4pw5 krb5kdc[21097](info): set up 2 sockets
Sep 26 11:39:46 dist-test-slave-4pw5 krb5kdc[21097](info): commencing operation
krb5kdc: starting...
W20260926 11:39:47.760161 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:47 dist-test-slave-4pw5 krb5kdc[21097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:47.903882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:48.028813 21110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:48.029191 21110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:48.029286 21110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:48.033667 21110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:48.033790 21110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:48.033834 21110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:48.033903 21110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:48.033947 21110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:48.040418 21110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:48.041728 21110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:48.042924 21110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:48.052811 21115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:48.052834 21116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:48.052876 21118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:48.053285 21110 server_base.cc:1034] running on GCE node
I20260926 11:39:48.053917 21110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:48.055154 21110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:48.056370 21110 hybrid_clock.cc:648] HybridClock initialized: now 1790422788056336 us; error 48 us; skew 500 ppm
Sep 26 11:39:48 dist-test-slave-4pw5 krb5kdc[21097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:48.060927 21110 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:48.061995 21110 webserver.cc:460] Webserver started at http://127.0.0.1:33113/ using document root <none> and password file <none>
I20260926 11:39:48.062655 21110 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:48.062758 21110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:48.062999 21110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:48.065335 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "1f3f181f71a7460e81213ac05856cc97"
format_stamp: "Formatted at 2026-09-26 11:39:48 on dist-test-slave-4pw5"
server_key: "f22b38a9cdfd4895446c989b2b386631"
server_key_iv: "905d03e0fea7e8cf605d0eb55426af48"
server_key_version: "encryptionkey@0"
I20260926 11:39:48.065958 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f3f181f71a7460e81213ac05856cc97"
format_stamp: "Formatted at 2026-09-26 11:39:48 on dist-test-slave-4pw5"
server_key: "f22b38a9cdfd4895446c989b2b386631"
server_key_iv: "905d03e0fea7e8cf605d0eb55426af48"
server_key_version: "encryptionkey@0"
I20260926 11:39:48.070580 21110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 11:39:48.073421 21125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:48.074908 21110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:39:48.075114 21110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "1f3f181f71a7460e81213ac05856cc97"
format_stamp: "Formatted at 2026-09-26 11:39:48 on dist-test-slave-4pw5"
server_key: "f22b38a9cdfd4895446c989b2b386631"
server_key_iv: "905d03e0fea7e8cf605d0eb55426af48"
server_key_version: "encryptionkey@0"
I20260926 11:39:48.075259 21110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:48.091372 21110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:48.095216 21110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:48.095489 21110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:48.104571 21110 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41225
I20260926 11:39:48.104964 21177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41225 every 8 connection(s)
I20260926 11:39:48.105800 21110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:48.109086 21178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:48.111660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 21110
I20260926 11:39:48.111817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:48.112141 392 external_mini_cluster.cc:1428] Setting key d8011283e7d762bf6e46b2b101124c1b
I20260926 11:39:48.115947 21178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97: Bootstrap starting.
I20260926 11:39:48.118470 21178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:48.119310 21178 log.cc:824] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:48.121950 21178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97: No bootstrap required, opened a new log
I20260926 11:39:48.122068 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:48.113751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59391 (local address 127.0.0.1:41225)
0926 11:39:48.114280 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:48.114297 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:48.114336 (+ 39us) server_negotiation.cc:407] Connection header received
0926 11:39:48.115036 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:48.115056 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:48.115444 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:48.115870 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:48.116309 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:48.117326 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:48.118301 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:48.118867 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:48.119042 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:48.119972 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:48.119977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:48.119991 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:48.120523 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:48.120593 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:48.120623 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:48.120731 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:48.120912 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 11:39:48.121133 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":150,"threads_started":1}
I20260926 11:39:48.125324 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:48.126344 21178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3f181f71a7460e81213ac05856cc97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260926 11:39:48.126627 21178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:48.126713 21178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3f181f71a7460e81213ac05856cc97, State: Initialized, Role: FOLLOWER
I20260926 11:39:48.127239 21178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3f181f71a7460e81213ac05856cc97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260926 11:39:48.127434 21178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:48.127521 21178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:48.127660 21178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:48.128782 21178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3f181f71a7460e81213ac05856cc97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260926 11:39:48.129204 21178 leader_election.cc:302] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f3f181f71a7460e81213ac05856cc97; no voters:
I20260926 11:39:48.129578 21178 leader_election.cc:288] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:48.129652 21188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:48.129866 21188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [term 1 LEADER]: Becoming Leader. State: Replica: 1f3f181f71a7460e81213ac05856cc97, State: Running, Role: LEADER
I20260926 11:39:48.130230 21188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3f181f71a7460e81213ac05856cc97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260926 11:39:48.131026 21178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:48.132330 21189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f3f181f71a7460e81213ac05856cc97. Latest consensus state: current_term: 1 leader_uuid: "1f3f181f71a7460e81213ac05856cc97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3f181f71a7460e81213ac05856cc97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } } }
I20260926 11:39:48.132467 21189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:48.132615 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:48.127761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41561 (local address 127.0.0.1:41225)
0926 11:39:48.127879 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:48.127884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:48.128061 (+ 177us) server_negotiation.cc:407] Connection header received
0926 11:39:48.128391 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:48.128395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:48.128455 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:48.128548 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:48.128827 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:48.129518 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:48.130625 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:48.130877 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:48.131006 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:48.132094 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:48.132098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:48.132117 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:48.132136 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:48.132168 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:48.132182 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:48.132236 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:48.132420 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 11:39:48.132485 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 11:39:48.133685 21190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f3f181f71a7460e81213ac05856cc97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3f181f71a7460e81213ac05856cc97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } } }
I20260926 11:39:48.134120 21190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97 [sys.catalog]: This master's current role is: LEADER
W20260926 11:39:48.136617 21204 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:39:48.136724 21204 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:39:48.136794 21197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:48.137501 21197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:48.144065 21197 catalog_manager.cc:1347] Generated new cluster ID: 5c6594a8df1d4be29fe30490069c06d5
I20260926 11:39:48.144184 21197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:48.162832 21197 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:48.164345 21197 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:48.173179 21197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f3f181f71a7460e81213ac05856cc97: Generated new TSK 0
I20260926 11:39:48.174156 21197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:48.185743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 21110
2026-09-26T11:39:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:48 dist-test-slave-4pw5 krb5kdc[21209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:48 dist-test-slave-4pw5 krb5kdc[21209](info): set up 2 sockets
Sep 26 11:39:48 dist-test-slave-4pw5 krb5kdc[21209](info): commencing operation
krb5kdc: starting...
W20260926 11:39:49.283861 392 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:49 dist-test-slave-4pw5 krb5kdc[21209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:49.424654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:49.542462 21222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:49.542855 21222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:49.542948 21222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:49.547715 21222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:49.547866 21222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:49.547909 21222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:49.547950 21222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:49.547986 21222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:49.554687 21222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:49.556183 21222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:49.558019 21222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:49.568244 21228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:49.568322 21227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:49.568244 21230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:49.568456 21222 server_base.cc:1034] running on GCE node
I20260926 11:39:49.569401 21222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:49.570698 21222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:49.571888 21222 hybrid_clock.cc:648] HybridClock initialized: now 1790422789571837 us; error 91 us; skew 500 ppm
Sep 26 11:39:49 dist-test-slave-4pw5 krb5kdc[21209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:49.574934 21222 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:49.576049 21222 webserver.cc:460] Webserver started at http://127.0.0.1:45155/ using document root <none> and password file <none>
I20260926 11:39:49.576651 21222 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:49.576745 21222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:49.576977 21222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:49.579114 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1"
format_stamp: "Formatted at 2026-09-26 11:39:49 on dist-test-slave-4pw5"
server_key: "08e606b01c5fb1063188295ebb126605"
server_key_iv: "cd46b3ed291f57abcdedbbb03f5eeddf"
server_key_version: "encryptionkey@0"
I20260926 11:39:49.579787 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1"
format_stamp: "Formatted at 2026-09-26 11:39:49 on dist-test-slave-4pw5"
server_key: "08e606b01c5fb1063188295ebb126605"
server_key_iv: "cd46b3ed291f57abcdedbbb03f5eeddf"
server_key_version: "encryptionkey@0"
I20260926 11:39:49.583896 21222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:49.586807 21237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:49.588501 21222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:49.588680 21222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1"
format_stamp: "Formatted at 2026-09-26 11:39:49 on dist-test-slave-4pw5"
server_key: "08e606b01c5fb1063188295ebb126605"
server_key_iv: "cd46b3ed291f57abcdedbbb03f5eeddf"
server_key_version: "encryptionkey@0"
I20260926 11:39:49.588837 21222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:49.594033 21222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:49.597663 21222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:49.597900 21222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:49.605693 21222 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37353
I20260926 11:39:49.605713 21289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37353 every 8 connection(s)
I20260926 11:39:49.606781 21222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:49.609701 21290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:49.612030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 21222
I20260926 11:39:49.612159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:49.612452 392 external_mini_cluster.cc:1428] Setting key 22cc2c9a36759b2c1ba2037491384c2f
I20260926 11:39:49.617005 21290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1: Bootstrap starting.
I20260926 11:39:49.618984 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:49.614089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42157 (local address 127.0.0.1:37353)
0926 11:39:49.614598 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:49.614618 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:49.614666 (+ 48us) server_negotiation.cc:407] Connection header received
0926 11:39:49.615608 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:49.615635 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:49.616015 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:49.616537 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:49.617441 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:49.617448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:49.617463 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:49.617477 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:49.617547 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:49.617577 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:49.617739 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:49.617824 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 11:39:49.617984 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":143,"threads_started":1}
I20260926 11:39:49.620175 21290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:49.621073 21290 log.cc:824] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:49.622098 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:49.623806 21290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1: No bootstrap required, opened a new log
I20260926 11:39:49.626689 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:49.624750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36471 (local address 127.0.0.1:37353)
0926 11:39:49.624910 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:49.624916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:49.625121 (+ 205us) server_negotiation.cc:407] Connection header received
0926 11:39:49.625265 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:49.625268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:49.625302 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:49.625376 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:49.626219 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:49.626223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:49.626225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:49.626237 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:49.626266 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:49.626277 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:49.626357 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:49.626480 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 11:39:49.626517 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:39:49.628942 21290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260926 11:39:49.629199 21290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:49.629279 21290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9a8247a5ec4b4ebf0d96c6df7149e1, State: Initialized, Role: FOLLOWER
I20260926 11:39:49.629741 21290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260926 11:39:49.629873 21290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:49.629967 21290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:49.630088 21290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:49.631322 21290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260926 11:39:49.631772 21290 leader_election.cc:302] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b9a8247a5ec4b4ebf0d96c6df7149e1; no voters:
I20260926 11:39:49.632138 21290 leader_election.cc:288] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:49.632349 21300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:49.632586 21300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [term 1 LEADER]: Becoming Leader. State: Replica: 9b9a8247a5ec4b4ebf0d96c6df7149e1, State: Running, Role: LEADER
I20260926 11:39:49.632930 21300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260926 11:39:49.633531 21290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:49.634730 21302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b9a8247a5ec4b4ebf0d96c6df7149e1. Latest consensus state: current_term: 1 leader_uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } } }
I20260926 11:39:49.634871 21302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:49.635272 21309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:49.635245 21301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a8247a5ec4b4ebf0d96c6df7149e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } } }
I20260926 11:39:49.635399 21301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:49.638952 21309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:49.644676 21309 catalog_manager.cc:1347] Generated new cluster ID: 85e29aa1679945668f075af021ba54c7
I20260926 11:39:49.644763 21309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:49.665119 21309 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:49.665508 21309 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:49.669525 21309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b9a8247a5ec4b4ebf0d96c6df7149e1: Generated new TSK 0
I20260926 11:39:49.670351 21309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:49.675108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 21222
2026-09-26T11:39:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:49 dist-test-slave-4pw5 krb5kdc[21321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:49 dist-test-slave-4pw5 krb5kdc[21321](info): set up 2 sockets
Sep 26 11:39:49 dist-test-slave-4pw5 krb5kdc[21321](info): commencing operation
krb5kdc: starting...
W20260926 11:39:50.748428 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:50 dist-test-slave-4pw5 krb5kdc[21321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:50.891963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:51.008011 21334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:51.008457 21334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:51.008529 21334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:51.013058 21334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:51.013139 21334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:51.013159 21334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:51.013177 21334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:51.013193 21334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:51.018958 21334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:51.020105 21334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:51.021112 21334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:51.030494 21342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:51.030488 21339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:51.030598 21334 server_base.cc:1034] running on GCE node
W20260926 11:39:51.030488 21340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:51.031459 21334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:51.032732 21334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:51.033926 21334 hybrid_clock.cc:648] HybridClock initialized: now 1790422791033912 us; error 32 us; skew 500 ppm
Sep 26 11:39:51 dist-test-slave-4pw5 krb5kdc[21321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:51.042984 21334 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:51.044260 21334 webserver.cc:460] Webserver started at http://127.0.0.1:34081/ using document root <none> and password file <none>
I20260926 11:39:51.044948 21334 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:51.045056 21334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:51.045324 21334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:51.047891 21334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "baa6c02a4b4b4e50905b3c5e53c011fd"
format_stamp: "Formatted at 2026-09-26 11:39:51 on dist-test-slave-4pw5"
server_key: "f78ff124006ece7f0fc45b4a75bb1d2b"
server_key_iv: "da3c7062c6387e9f903d905130f2ab1d"
server_key_version: "encryptionkey@0"
I20260926 11:39:51.048595 21334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "baa6c02a4b4b4e50905b3c5e53c011fd"
format_stamp: "Formatted at 2026-09-26 11:39:51 on dist-test-slave-4pw5"
server_key: "f78ff124006ece7f0fc45b4a75bb1d2b"
server_key_iv: "da3c7062c6387e9f903d905130f2ab1d"
server_key_version: "encryptionkey@0"
I20260926 11:39:51.053136 21334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:51.056407 21349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:51.058030 21334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:39:51.058233 21334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "baa6c02a4b4b4e50905b3c5e53c011fd"
format_stamp: "Formatted at 2026-09-26 11:39:51 on dist-test-slave-4pw5"
server_key: "f78ff124006ece7f0fc45b4a75bb1d2b"
server_key_iv: "da3c7062c6387e9f903d905130f2ab1d"
server_key_version: "encryptionkey@0"
I20260926 11:39:51.058390 21334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:51.086715 21334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:51.090813 21334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:51.091078 21334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:51.099017 21334 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35655
I20260926 11:39:51.099033 21401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35655 every 8 connection(s)
I20260926 11:39:51.100134 21334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:51.103404 21402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:51.109378 21402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd: Bootstrap starting.
I20260926 11:39:51.109777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 21334
I20260926 11:39:51.109922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:51.110209 392 external_mini_cluster.cc:1428] Setting key dda5db0e2a44e45525ee71605f913701
I20260926 11:39:51.112411 21402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:51.113222 21402 log.cc:824] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:51.115437 21402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd: No bootstrap required, opened a new log
Sep 26 11:39:51 dist-test-slave-4pw5 krb5kdc[21321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 11:39:51.120458 21402 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa6c02a4b4b4e50905b3c5e53c011fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35655 } }
I20260926 11:39:51.120754 21402 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:51.120833 21402 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa6c02a4b4b4e50905b3c5e53c011fd, State: Initialized, Role: FOLLOWER
I20260926 11:39:51.121361 21402 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa6c02a4b4b4e50905b3c5e53c011fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35655 } }
I20260926 11:39:51.121552 21402 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:51.121644 21402 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:51.121759 21402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:51.122879 21402 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa6c02a4b4b4e50905b3c5e53c011fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35655 } }
I20260926 11:39:51.123258 21402 leader_election.cc:302] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa6c02a4b4b4e50905b3c5e53c011fd; no voters:
I20260926 11:39:51.123797 21402 leader_election.cc:288] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:51.123924 21407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:51.124214 21407 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [term 1 LEADER]: Becoming Leader. State: Replica: baa6c02a4b4b4e50905b3c5e53c011fd, State: Running, Role: LEADER
I20260926 11:39:51.124627 21407 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa6c02a4b4b4e50905b3c5e53c011fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35655 } }
I20260926 11:39:51.125202 21402 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:51.126128 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa6c02a4b4b4e50905b3c5e53c011fd. Latest consensus state: current_term: 1 leader_uuid: "baa6c02a4b4b4e50905b3c5e53c011fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa6c02a4b4b4e50905b3c5e53c011fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35655 } } }
I20260926 11:39:51.126363 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:51.126411 21408 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa6c02a4b4b4e50905b3c5e53c011fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa6c02a4b4b4e50905b3c5e53c011fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35655 } } }
I20260926 11:39:51.126600 21408 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:51.126806 21416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:51.126925 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:51.111611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:35987 (local address 127.0.0.1:35655)
0926 11:39:51.112073 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:51.112088 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:51.112126 (+ 38us) server_negotiation.cc:407] Connection header received
0926 11:39:51.113054 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:51.113075 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:51.113502 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:51.113856 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:51.114908 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:51.116008 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:51.117237 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:51.117771 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:51.117900 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:51.121103 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:51.121108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:51.121123 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:51.121157 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:51.123075 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:51.123704 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:51.123711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:51.123718 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:51.123943 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:51.124423 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:51.124426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:51.124428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:51.124982 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:51.125222 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:51.125541 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 11:39:51.125796 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":136,"threads_started":1}
I20260926 11:39:51.130472 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:51.131155 21416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:51.137531 21416 catalog_manager.cc:1347] Generated new cluster ID: 87f69b46ff66411c95ec6c821a0d0aa7
I20260926 11:39:51.137708 21416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:51.143352 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:51.132919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:56987 (local address 127.0.0.1:35655)
0926 11:39:51.133033 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:51.133038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:51.133340 (+ 302us) server_negotiation.cc:407] Connection header received
0926 11:39:51.133472 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:51.133478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:51.133534 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:51.133629 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:51.134473 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:51.135747 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:51.137204 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:51.137670 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:51.137776 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:51.139568 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:51.139601 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:51.139604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:51.139635 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:51.141405 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:51.141968 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:51.141972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:51.141974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:51.142069 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:51.142399 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:51.142402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:51.142404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:51.142637 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:51.142788 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:51.143049 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 11:39:51.143154 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:39:51.151925 21416 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:51.153554 21416 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:51.160486 21416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa6c02a4b4b4e50905b3c5e53c011fd: Generated new TSK 0
I20260926 11:39:51.161449 21416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:51.174053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 21334
W20260926 11:39:51.179677 21319 connection.cc:535] client connection to 127.0.0.1:35655 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35655): Connection reset by peer (error 104)
2026-09-26T11:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:51 dist-test-slave-4pw5 krb5kdc[21433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:51 dist-test-slave-4pw5 krb5kdc[21433](info): set up 2 sockets
Sep 26 11:39:51 dist-test-slave-4pw5 krb5kdc[21433](info): commencing operation
krb5kdc: starting...
W20260926 11:39:52.272055 392 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:52 dist-test-slave-4pw5 krb5kdc[21433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:52.411926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:52.541937 21446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:52.542266 21446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:52.542313 21446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:52.547001 21446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:52.547102 21446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:52.547124 21446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:52.547142 21446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:52.547158 21446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:52.554046 21446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:52.555410 21446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:52.556604 21446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:52.566496 21454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:52.566502 21452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:52.566648 21446 server_base.cc:1034] running on GCE node
W20260926 11:39:52.566864 21451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:52.567521 21446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:52.568792 21446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:52.569962 21446 hybrid_clock.cc:648] HybridClock initialized: now 1790422792569918 us; error 62 us; skew 500 ppm
Sep 26 11:39:52 dist-test-slave-4pw5 krb5kdc[21433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:52.573206 21446 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:52.574345 21446 webserver.cc:460] Webserver started at http://127.0.0.1:42739/ using document root <none> and password file <none>
I20260926 11:39:52.574988 21446 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:52.575058 21446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:52.575305 21446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:52.577745 21446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "21a404412722409cb5c318d1bc313a7a"
format_stamp: "Formatted at 2026-09-26 11:39:52 on dist-test-slave-4pw5"
server_key: "9826fef6865de1cf92cde5bf66d02797"
server_key_iv: "00b248b2dea053d1a7fabd072b52b499"
server_key_version: "encryptionkey@0"
I20260926 11:39:52.578413 21446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "21a404412722409cb5c318d1bc313a7a"
format_stamp: "Formatted at 2026-09-26 11:39:52 on dist-test-slave-4pw5"
server_key: "9826fef6865de1cf92cde5bf66d02797"
server_key_iv: "00b248b2dea053d1a7fabd072b52b499"
server_key_version: "encryptionkey@0"
I20260926 11:39:52.582819 21446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 11:39:52.585794 21461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:52.587286 21446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:39:52.587502 21446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "21a404412722409cb5c318d1bc313a7a"
format_stamp: "Formatted at 2026-09-26 11:39:52 on dist-test-slave-4pw5"
server_key: "9826fef6865de1cf92cde5bf66d02797"
server_key_iv: "00b248b2dea053d1a7fabd072b52b499"
server_key_version: "encryptionkey@0"
I20260926 11:39:52.587715 21446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:52.627476 21446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:52.631121 21446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:52.631304 21446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:52.639793 21446 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41499
I20260926 11:39:52.639812 21513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41499 every 8 connection(s)
I20260926 11:39:52.641148 21446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:52.644471 21514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:52.650866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 21446
I20260926 11:39:52.651012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:52.651520 392 external_mini_cluster.cc:1428] Setting key b20cd4dcac77cbe5b8e7cf954cfa0dbd
I20260926 11:39:52.651592 21514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a: Bootstrap starting.
I20260926 11:39:52.655118 21514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:52.656139 21514 log.cc:824] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:52.658375 21514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a: No bootstrap required, opened a new log
I20260926 11:39:52.662935 21514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a404412722409cb5c318d1bc313a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } }
I20260926 11:39:52.663245 21514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:52.663347 21514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a404412722409cb5c318d1bc313a7a, State: Initialized, Role: FOLLOWER
I20260926 11:39:52.663431 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:52.653497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:36747 (local address 127.0.0.1:41499)
0926 11:39:52.654326 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:52.654349 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:52.654407 (+ 58us) server_negotiation.cc:407] Connection header received
0926 11:39:52.655540 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:52.655564 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:52.656052 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:52.656442 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:52.656919 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:52.658254 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:52.659392 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:52.660026 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:52.660236 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:52.661466 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:52.661472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:52.661488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:52.661937 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:52.662008 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:52.662033 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:52.662184 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:52.662381 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 11:39:52.662551 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":170,"threads_started":1}
I20260926 11:39:52.663978 21514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a404412722409cb5c318d1bc313a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } }
I20260926 11:39:52.664166 21514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:52.664259 21514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:52.664366 21514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:52.665544 21514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a404412722409cb5c318d1bc313a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } }
I20260926 11:39:52.665983 21514 leader_election.cc:302] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21a404412722409cb5c318d1bc313a7a; no voters:
I20260926 11:39:52.666399 21514 leader_election.cc:288] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:52.666551 21519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:52.666839 21519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [term 1 LEADER]: Becoming Leader. State: Replica: 21a404412722409cb5c318d1bc313a7a, State: Running, Role: LEADER
I20260926 11:39:52.667289 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:52.667224 21519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a404412722409cb5c318d1bc313a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } }
I20260926 11:39:52.668066 21514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:52.670176 21520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21a404412722409cb5c318d1bc313a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a404412722409cb5c318d1bc313a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } } }
I20260926 11:39:52.670347 21520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:52.670720 21521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21a404412722409cb5c318d1bc313a7a. Latest consensus state: current_term: 1 leader_uuid: "21a404412722409cb5c318d1bc313a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a404412722409cb5c318d1bc313a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41499 } } }
I20260926 11:39:52.670876 21521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a [sys.catalog]: This master's current role is: LEADER
W20260926 11:39:52.674286 21538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:39:52.674409 21538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:39:52.674497 21539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:52.675562 21539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:52.677127 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:52.670334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:56437 (local address 127.0.0.1:41499)
0926 11:39:52.671006 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:52.671012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:52.671958 (+ 946us) server_negotiation.cc:407] Connection header received
0926 11:39:52.672164 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:52.672167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:52.672218 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:52.672454 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:52.672797 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:52.673647 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:52.674909 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:52.675266 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:52.675406 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:52.676565 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:52.676569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:52.676572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:52.676590 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:52.676623 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:52.676637 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:52.676761 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:52.676949 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 11:39:52.676989 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572}
I20260926 11:39:52.682984 21539 catalog_manager.cc:1347] Generated new cluster ID: b777f864b2ce4339ad824858348854c3
I20260926 11:39:52.683077 21539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:52.689947 21539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:52.691052 21539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:52.702417 21539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21a404412722409cb5c318d1bc313a7a: Generated new TSK 0
I20260926 11:39:52.703400 21539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:52.713630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 21446
W20260926 11:39:52.718477 21431 connection.cc:535] client connection to 127.0.0.1:41499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41499): Connection reset by peer (error 104)
2026-09-26T11:39:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1537 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:52 dist-test-slave-4pw5 krb5kdc[21545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:52 dist-test-slave-4pw5 krb5kdc[21545](info): set up 2 sockets
Sep 26 11:39:52 dist-test-slave-4pw5 krb5kdc[21545](info): commencing operation
krb5kdc: starting...
W20260926 11:39:53.816409 392 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:53 dist-test-slave-4pw5 krb5kdc[21545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:53.968293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:54.108949 21558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:54.109362 21558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:54.109412 21558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:54.114423 21558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:54.114557 21558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:54.114578 21558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:54.114598 21558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:54.114615 21558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:54.121367 21558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:54.123039 21558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:54.124467 21558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:54.135828 21566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:54.135913 21558 server_base.cc:1034] running on GCE node
W20260926 11:39:54.135859 21563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:54.135828 21564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:54.137116 21558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:54.138589 21558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:54.139905 21558 hybrid_clock.cc:648] HybridClock initialized: now 1790422794139886 us; error 102 us; skew 500 ppm
Sep 26 11:39:54 dist-test-slave-4pw5 krb5kdc[21545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:54.143831 21558 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:54.145154 21558 webserver.cc:460] Webserver started at http://127.0.0.1:42389/ using document root <none> and password file <none>
I20260926 11:39:54.145884 21558 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:54.145984 21558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:54.146210 21558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:54.149024 21558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "074c963d0c634c228e8fa7d9149d125f"
format_stamp: "Formatted at 2026-09-26 11:39:54 on dist-test-slave-4pw5"
server_key: "2cde3ee9dfa877f82709b133cd578bf5"
server_key_iv: "3049b995d59fb9601d83670127c540e4"
server_key_version: "encryptionkey@0"
I20260926 11:39:54.149735 21558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "074c963d0c634c228e8fa7d9149d125f"
format_stamp: "Formatted at 2026-09-26 11:39:54 on dist-test-slave-4pw5"
server_key: "2cde3ee9dfa877f82709b133cd578bf5"
server_key_iv: "3049b995d59fb9601d83670127c540e4"
server_key_version: "encryptionkey@0"
I20260926 11:39:54.154824 21558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 11:39:54.158448 21573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:54.160331 21558 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:39:54.160531 21558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "074c963d0c634c228e8fa7d9149d125f"
format_stamp: "Formatted at 2026-09-26 11:39:54 on dist-test-slave-4pw5"
server_key: "2cde3ee9dfa877f82709b133cd578bf5"
server_key_iv: "3049b995d59fb9601d83670127c540e4"
server_key_version: "encryptionkey@0"
I20260926 11:39:54.160688 21558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:54.166743 21558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:54.170746 21558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:54.171005 21558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:54.180619 21558 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35493
I20260926 11:39:54.180728 21625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35493 every 8 connection(s)
I20260926 11:39:54.181931 21558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:54.185813 21626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:54.187312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 21558
I20260926 11:39:54.187491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:54.187898 392 external_mini_cluster.cc:1428] Setting key 06f414c3f5825dd20d239b19e77da1df
I20260926 11:39:54.194223 21626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f: Bootstrap starting.
I20260926 11:39:54.195497 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:54.189784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:46933 (local address 127.0.0.1:35493)
0926 11:39:54.190384 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:54.190407 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:54.190459 (+ 52us) server_negotiation.cc:407] Connection header received
0926 11:39:54.191413 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:54.191441 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:54.191901 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:54.192384 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:54.193812 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:54.193819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:54.193836 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:54.193850 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:54.193929 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:54.193960 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:54.194148 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:54.194283 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 11:39:54.194468 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":171,"threads_started":1}
I20260926 11:39:54.197430 21626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:54.198441 21626 log.cc:824] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:54.199687 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:54.201704 21626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f: No bootstrap required, opened a new log
I20260926 11:39:54.204938 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:54.202496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:38543 (local address 127.0.0.1:35493)
0926 11:39:54.202654 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:54.202659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:54.202966 (+ 307us) server_negotiation.cc:407] Connection header received
0926 11:39:54.203104 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:54.203107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:54.203158 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:54.203292 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:54.204487 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:54.204491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:54.204494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:54.204508 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:54.204542 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:54.204555 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:54.204673 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:54.204751 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 11:39:54.204801 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 11:39:54.206709 21626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074c963d0c634c228e8fa7d9149d125f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260926 11:39:54.206981 21626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:54.207016 21626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074c963d0c634c228e8fa7d9149d125f, State: Initialized, Role: FOLLOWER
I20260926 11:39:54.207485 21626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074c963d0c634c228e8fa7d9149d125f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260926 11:39:54.207687 21626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:54.207758 21626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:54.207844 21626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:54.208833 21626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074c963d0c634c228e8fa7d9149d125f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260926 11:39:54.209256 21626 leader_election.cc:302] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 074c963d0c634c228e8fa7d9149d125f; no voters:
I20260926 11:39:54.209704 21626 leader_election.cc:288] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:54.209870 21636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:54.210218 21636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [term 1 LEADER]: Becoming Leader. State: Replica: 074c963d0c634c228e8fa7d9149d125f, State: Running, Role: LEADER
I20260926 11:39:54.210736 21636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074c963d0c634c228e8fa7d9149d125f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260926 11:39:54.211184 21626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:54.213330 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 074c963d0c634c228e8fa7d9149d125f. Latest consensus state: current_term: 1 leader_uuid: "074c963d0c634c228e8fa7d9149d125f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074c963d0c634c228e8fa7d9149d125f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } } }
I20260926 11:39:54.213286 21637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "074c963d0c634c228e8fa7d9149d125f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074c963d0c634c228e8fa7d9149d125f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } } }
I20260926 11:39:54.213493 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:54.213500 21637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f [sys.catalog]: This master's current role is: LEADER
W20260926 11:39:54.217985 21651 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:39:54.218084 21651 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:39:54.218183 21649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:54.218976 21649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:54.225841 21649 catalog_manager.cc:1347] Generated new cluster ID: 79e23e0eb44a4c3fae45104c25eac609
I20260926 11:39:54.225965 21649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:54.254040 21649 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:54.254549 21649 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:54.270637 21649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 074c963d0c634c228e8fa7d9149d125f: Generated new TSK 0
I20260926 11:39:54.271649 21649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:54.276237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 21558
2026-09-26T11:39:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1562 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:54 dist-test-slave-4pw5 krb5kdc[21657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:54 dist-test-slave-4pw5 krb5kdc[21657](info): set up 2 sockets
Sep 26 11:39:54 dist-test-slave-4pw5 krb5kdc[21657](info): commencing operation
krb5kdc: starting...
W20260926 11:39:55.368050 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:55 dist-test-slave-4pw5 krb5kdc[21657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:55.503038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:55.632097 21670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:55.632462 21670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:55.632550 21670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:55.636817 21670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:55.636940 21670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:55.636977 21670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:55.637019 21670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:55.637051 21670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:55.642772 21670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39391
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:55.644049 21670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:55.645200 21670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:55.654452 21676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:55.654452 21675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:55.654507 21678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:55.654685 21670 server_base.cc:1034] running on GCE node
I20260926 11:39:55.655417 21670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:55.656692 21670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:55.657918 21670 hybrid_clock.cc:648] HybridClock initialized: now 1790422795657887 us; error 47 us; skew 500 ppm
Sep 26 11:39:55 dist-test-slave-4pw5 krb5kdc[21657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:55.661303 21670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:55.662369 21670 webserver.cc:460] Webserver started at http://127.0.0.1:43841/ using document root <none> and password file <none>
I20260926 11:39:55.663041 21670 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:55.663141 21670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:55.663398 21670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:55.665647 21670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "7288afc779e541fa9d89d37ca0c2972d"
format_stamp: "Formatted at 2026-09-26 11:39:55 on dist-test-slave-4pw5"
server_key: "a5851f601bad1f3e03b42a7e256066c5"
server_key_iv: "8d31279e915e99cf09e1ceac7d5298fe"
server_key_version: "encryptionkey@0"
I20260926 11:39:55.666271 21670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "7288afc779e541fa9d89d37ca0c2972d"
format_stamp: "Formatted at 2026-09-26 11:39:55 on dist-test-slave-4pw5"
server_key: "a5851f601bad1f3e03b42a7e256066c5"
server_key_iv: "8d31279e915e99cf09e1ceac7d5298fe"
server_key_version: "encryptionkey@0"
I20260926 11:39:55.670390 21670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:39:55.673281 21685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:55.674731 21670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 11:39:55.674897 21670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "7288afc779e541fa9d89d37ca0c2972d"
format_stamp: "Formatted at 2026-09-26 11:39:55 on dist-test-slave-4pw5"
server_key: "a5851f601bad1f3e03b42a7e256066c5"
server_key_iv: "8d31279e915e99cf09e1ceac7d5298fe"
server_key_version: "encryptionkey@0"
I20260926 11:39:55.675040 21670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:55.713376 21670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:55.717023 21670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:55.717258 21670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:55.724882 21670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39391
I20260926 11:39:55.724897 21737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39391 every 8 connection(s)
I20260926 11:39:55.725960 21670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:55.729151 21738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:55.730986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 21670
I20260926 11:39:55.731141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:55.731422 392 external_mini_cluster.cc:1428] Setting key 8faf354a31873514299e00540f4a4cef
I20260926 11:39:55.736006 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d: Bootstrap starting.
I20260926 11:39:55.738709 21738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:55.739835 21738 log.cc:824] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:39:55 dist-test-slave-4pw5 krb5kdc[21657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 11:39:55.742479 21738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d: No bootstrap required, opened a new log
I20260926 11:39:55.748212 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:55.732912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:33929 (local address 127.0.0.1:39391)
0926 11:39:55.733357 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:55.733379 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:55.733426 (+ 47us) server_negotiation.cc:407] Connection header received
0926 11:39:55.734069 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:55.734085 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:55.734426 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:55.734717 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:55.735723 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:55.736805 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:55.738013 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:55.738573 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:55.738735 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:55.742068 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:55.742075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:55.742092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:55.742123 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:55.744421 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:55.745076 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:55.745085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:55.745095 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:55.745297 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:55.745655 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:55.745661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:55.745663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:55.746283 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:55.746522 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:55.746770 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 11:39:55.747042 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":146,"threads_started":1}
I20260926 11:39:55.748081 21738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7288afc779e541fa9d89d37ca0c2972d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260926 11:39:55.748495 21738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:55.748623 21738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7288afc779e541fa9d89d37ca0c2972d, State: Initialized, Role: FOLLOWER
I20260926 11:39:55.749514 21738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7288afc779e541fa9d89d37ca0c2972d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260926 11:39:55.749708 21738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:55.749800 21738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:55.749933 21738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:55.751191 21738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7288afc779e541fa9d89d37ca0c2972d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260926 11:39:55.751662 21738 leader_election.cc:302] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7288afc779e541fa9d89d37ca0c2972d; no voters:
I20260926 11:39:55.752060 21738 leader_election.cc:288] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:55.752141 21743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:55.752192 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:55.752319 21743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [term 1 LEADER]: Becoming Leader. State: Replica: 7288afc779e541fa9d89d37ca0c2972d, State: Running, Role: LEADER
I20260926 11:39:55.752638 21743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7288afc779e541fa9d89d37ca0c2972d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260926 11:39:55.753463 21738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:55.753931 21743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7288afc779e541fa9d89d37ca0c2972d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7288afc779e541fa9d89d37ca0c2972d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } } }
I20260926 11:39:55.754109 21743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:55.754346 21743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7288afc779e541fa9d89d37ca0c2972d. Latest consensus state: current_term: 1 leader_uuid: "7288afc779e541fa9d89d37ca0c2972d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7288afc779e541fa9d89d37ca0c2972d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } } }
I20260926 11:39:55.754506 21743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:55.754822 21752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:55.759120 21752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:55.764389 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:55.755028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:36983 (local address 127.0.0.1:39391)
0926 11:39:55.755152 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:55.755157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:55.755170 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:39:55.755216 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:55.755219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:55.755281 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:55.755366 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:55.756368 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:55.757099 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:55.758014 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:55.758345 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:55.758460 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:55.760713 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:55.760717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:55.760720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:39:55.760749 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:55.762448 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:55.762985 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:55.762989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:55.762991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:55.763075 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:39:55.763416 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:39:55.763420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:39:55.763421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:39:55.763683 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:55.763846 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:55.764114 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 11:39:55.764214 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:39:55.766693 21752 catalog_manager.cc:1347] Generated new cluster ID: e203da734d3d46f089f349f44aaad56c
I20260926 11:39:55.766762 21752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:55.791666 21752 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:55.793006 21752 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:55.802282 21752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7288afc779e541fa9d89d37ca0c2972d: Generated new TSK 0
I20260926 11:39:55.803228 21752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:55.808036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 21670
W20260926 11:39:55.813750 21655 connection.cc:535] client connection to 127.0.0.1:39391 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39391): Connection reset by peer (error 104)
2026-09-26T11:39:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:55 dist-test-slave-4pw5 krb5kdc[21769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:55 dist-test-slave-4pw5 krb5kdc[21769](info): set up 2 sockets
Sep 26 11:39:55 dist-test-slave-4pw5 krb5kdc[21769](info): commencing operation
krb5kdc: starting...
W20260926 11:39:56.899960 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:56 dist-test-slave-4pw5 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:57.037186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:57.158543 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:57.158844 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:57.158887 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:57.163070 21782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:57.163166 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:57.163194 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:57.163252 21782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:57.163321 21782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:57.168977 21782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:57.170158 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:57.171234 21782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:57.180207 21788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:57.180287 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:57.180323 21787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:57.180599 21782 server_base.cc:1034] running on GCE node
I20260926 11:39:57.181604 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:57.182816 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:57.184011 21782 hybrid_clock.cc:648] HybridClock initialized: now 1790422797183987 us; error 42 us; skew 500 ppm
Sep 26 11:39:57 dist-test-slave-4pw5 krb5kdc[21769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:57.193295 21782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:57.194916 21782 webserver.cc:460] Webserver started at http://127.0.0.1:35797/ using document root <none> and password file <none>
I20260926 11:39:57.195643 21782 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:57.195760 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:57.195989 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:57.198689 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "5e96d12540254fa099b788c14b4cb0bf"
format_stamp: "Formatted at 2026-09-26 11:39:57 on dist-test-slave-4pw5"
server_key: "db01ff9babcc595734f5c75aacae9e92"
server_key_iv: "487568b015833618fde152d063ece231"
server_key_version: "encryptionkey@0"
I20260926 11:39:57.199687 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e96d12540254fa099b788c14b4cb0bf"
format_stamp: "Formatted at 2026-09-26 11:39:57 on dist-test-slave-4pw5"
server_key: "db01ff9babcc595734f5c75aacae9e92"
server_key_iv: "487568b015833618fde152d063ece231"
server_key_version: "encryptionkey@0"
I20260926 11:39:57.205458 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 11:39:57.208359 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:57.209887 21782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:57.210022 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "5e96d12540254fa099b788c14b4cb0bf"
format_stamp: "Formatted at 2026-09-26 11:39:57 on dist-test-slave-4pw5"
server_key: "db01ff9babcc595734f5c75aacae9e92"
server_key_iv: "487568b015833618fde152d063ece231"
server_key_version: "encryptionkey@0"
I20260926 11:39:57.210143 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:57.232206 21782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:57.235687 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:57.235862 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:57.243463 21782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35755
I20260926 11:39:57.243620 21849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35755 every 8 connection(s)
I20260926 11:39:57.244554 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:57.245375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 21782
I20260926 11:39:57.245518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:57.245814 392 external_mini_cluster.cc:1428] Setting key f12bd5b181e6737d1edfed708684b4b8
I20260926 11:39:57.248241 21850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:57.255563 21850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf: Bootstrap starting.
I20260926 11:39:57.255921 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:57.247209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:44363 (local address 127.0.0.1:35755)
0926 11:39:57.247671 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:57.247688 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:57.247723 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:39:57.248495 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:57.248514 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:57.248863 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:57.249161 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:57.249609 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:57.250693 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:57.252117 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:57.252673 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:57.252806 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:57.253885 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:57.253890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:57.253903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:57.254361 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:57.254422 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:57.254446 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:57.254551 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:57.254783 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 11:39:57.254972 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":146,"threads_started":1}
I20260926 11:39:57.258301 21850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:57.259228 21850 log.cc:824] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:57.259262 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:57.261798 21850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf: No bootstrap required, opened a new log
I20260926 11:39:57.265571 21850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e96d12540254fa099b788c14b4cb0bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260926 11:39:57.265856 21850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:57.265990 21850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e96d12540254fa099b788c14b4cb0bf, State: Initialized, Role: FOLLOWER
I20260926 11:39:57.266140 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:57.261905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:52123 (local address 127.0.0.1:35755)
0926 11:39:57.262023 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:57.262035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:57.262205 (+ 170us) server_negotiation.cc:407] Connection header received
0926 11:39:57.262374 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:57.262378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:57.262427 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:57.262513 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:57.262798 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:57.263378 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:57.264315 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:39:57.264543 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:39:57.264650 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:39:57.265567 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:57.265577 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:57.265580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:57.265592 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:57.265618 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:57.265629 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:57.265728 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:57.265940 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 11:39:57.265988 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 11:39:57.266501 21850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e96d12540254fa099b788c14b4cb0bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260926 11:39:57.266680 21850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:57.266762 21850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:57.266906 21850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:57.268116 21850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e96d12540254fa099b788c14b4cb0bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260926 11:39:57.268530 21850 leader_election.cc:302] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e96d12540254fa099b788c14b4cb0bf; no voters:
I20260926 11:39:57.268931 21850 leader_election.cc:288] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:57.269098 21860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:57.269335 21860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [term 1 LEADER]: Becoming Leader. State: Replica: 5e96d12540254fa099b788c14b4cb0bf, State: Running, Role: LEADER
I20260926 11:39:57.269694 21860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e96d12540254fa099b788c14b4cb0bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260926 11:39:57.270460 21850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:57.271857 21861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e96d12540254fa099b788c14b4cb0bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e96d12540254fa099b788c14b4cb0bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } } }
I20260926 11:39:57.271869 21862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e96d12540254fa099b788c14b4cb0bf. Latest consensus state: current_term: 1 leader_uuid: "5e96d12540254fa099b788c14b4cb0bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e96d12540254fa099b788c14b4cb0bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } } }
I20260926 11:39:57.272022 21861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:57.272025 21862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:57.273110 21870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:57.276768 21870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:57.282797 21870 catalog_manager.cc:1347] Generated new cluster ID: 63b72bb5e09b459ba874ca33bd8de03b
I20260926 11:39:57.282895 21870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:57.293629 21870 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:57.294746 21870 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:57.299764 21870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e96d12540254fa099b788c14b4cb0bf: Generated new TSK 0
I20260926 11:39:57.300840 21870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:57.306221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 21782
W20260926 11:39:57.311072 21767 connection.cc:535] client connection to 127.0.0.1:35755 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35755): Connection reset by peer (error 104)
2026-09-26T11:39:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1497 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:57 dist-test-slave-4pw5 krb5kdc[21881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:57 dist-test-slave-4pw5 krb5kdc[21881](info): set up 2 sockets
Sep 26 11:39:57 dist-test-slave-4pw5 krb5kdc[21881](info): commencing operation
krb5kdc: starting...
W20260926 11:39:58.360081 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:58 dist-test-slave-4pw5 krb5kdc[21881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:39:58.496104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:39:58.613106 21894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:39:58.613482 21894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:39:58.613577 21894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:39:58.617897 21894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:39:58.618017 21894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:39:58.618055 21894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:39:58.618095 21894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:39:58.618131 21894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:39:58.623912 21894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:39:58.625144 21894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:39:58.626271 21894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:39:58.635865 21899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:58.635865 21902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:39:58.635905 21900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:39:58.636245 21894 server_base.cc:1034] running on GCE node
I20260926 11:39:58.636896 21894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:39:58.638157 21894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:39:58.639370 21894 hybrid_clock.cc:648] HybridClock initialized: now 1790422798639358 us; error 32 us; skew 500 ppm
Sep 26 11:39:58 dist-test-slave-4pw5 krb5kdc[21881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:39:58.642649 21894 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:39:58.643837 21894 webserver.cc:460] Webserver started at http://127.0.0.1:41269/ using document root <none> and password file <none>
I20260926 11:39:58.644461 21894 fs_manager.cc:359] Metadata directory not provided
I20260926 11:39:58.644558 21894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:39:58.644802 21894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:39:58.647039 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "ab0c4c70c25443539bfad78327c28024"
format_stamp: "Formatted at 2026-09-26 11:39:58 on dist-test-slave-4pw5"
server_key: "ca8d556ce0043c4f6a2687272c381e84"
server_key_iv: "20e79250d0f08e8b8c4cfcba5b58da63"
server_key_version: "encryptionkey@0"
I20260926 11:39:58.647723 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab0c4c70c25443539bfad78327c28024"
format_stamp: "Formatted at 2026-09-26 11:39:58 on dist-test-slave-4pw5"
server_key: "ca8d556ce0043c4f6a2687272c381e84"
server_key_iv: "20e79250d0f08e8b8c4cfcba5b58da63"
server_key_version: "encryptionkey@0"
I20260926 11:39:58.651866 21894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:39:58.654567 21909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:39:58.655985 21894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:39:58.656137 21894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "ab0c4c70c25443539bfad78327c28024"
format_stamp: "Formatted at 2026-09-26 11:39:58 on dist-test-slave-4pw5"
server_key: "ca8d556ce0043c4f6a2687272c381e84"
server_key_iv: "20e79250d0f08e8b8c4cfcba5b58da63"
server_key_version: "encryptionkey@0"
I20260926 11:39:58.656270 21894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:39:58.661084 21894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:39:58.664513 21894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:39:58.664716 21894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:39:58.672403 21894 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43561
I20260926 11:39:58.672418 21961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43561 every 8 connection(s)
I20260926 11:39:58.673496 21894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:39:58.676539 21962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:39:58.682847 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024: Bootstrap starting.
I20260926 11:39:58.683475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 21894
I20260926 11:39:58.683660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:39:58.683966 392 external_mini_cluster.cc:1428] Setting key e0a77f46ca2e1665400cad0d061234ae
I20260926 11:39:58.686161 21962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024: Neither blocks nor log segments found. Creating new log.
I20260926 11:39:58.687180 21962 log.cc:824] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024: Log is configured to *not* fsync() on all Append() calls
I20260926 11:39:58.689882 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024: No bootstrap required, opened a new log
I20260926 11:39:58.690590 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:58.685586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:57613 (local address 127.0.0.1:43561)
0926 11:39:58.686205 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:58.686241 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:58.686290 (+ 49us) server_negotiation.cc:407] Connection header received
0926 11:39:58.687326 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:58.687353 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:58.687783 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:58.688146 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:58.688995 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:58.689000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:58.689015 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:58.689026 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:58.689085 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:58.689110 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:58.689241 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:58.689407 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 11:39:58.689577 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":190,"threads_started":1}
I20260926 11:39:58.693835 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:39:58.694416 21962 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c4c70c25443539bfad78327c28024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } }
I20260926 11:39:58.694656 21962 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:39:58.694716 21962 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0c4c70c25443539bfad78327c28024, State: Initialized, Role: FOLLOWER
I20260926 11:39:58.695161 21962 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c4c70c25443539bfad78327c28024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } }
I20260926 11:39:58.695359 21962 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:39:58.695498 21962 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:39:58.695654 21962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:39:58.696971 21962 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c4c70c25443539bfad78327c28024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } }
I20260926 11:39:58.697371 21962 leader_election.cc:302] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab0c4c70c25443539bfad78327c28024; no voters:
I20260926 11:39:58.697737 21962 leader_election.cc:288] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:39:58.697856 21972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:39:58.698119 21972 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [term 1 LEADER]: Becoming Leader. State: Replica: ab0c4c70c25443539bfad78327c28024, State: Running, Role: LEADER
I20260926 11:39:58.698452 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:39:58.696262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:36841 (local address 127.0.0.1:43561)
0926 11:39:58.696472 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 11:39:58.696476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:39:58.696737 (+ 261us) server_negotiation.cc:407] Connection header received
0926 11:39:58.696822 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:39:58.696828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:39:58.696876 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:39:58.696971 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:39:58.697952 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:39:58.697959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:39:58.697964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:39:58.697982 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:39:58.698017 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:39:58.698032 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:39:58.698126 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 11:39:58.698276 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 11:39:58.698320 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:39:58.698606 21972 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c4c70c25443539bfad78327c28024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } }
I20260926 11:39:58.699268 21962 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:39:58.700786 21973 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab0c4c70c25443539bfad78327c28024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c4c70c25443539bfad78327c28024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } } }
I20260926 11:39:58.700961 21973 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [sys.catalog]: This master's current role is: LEADER
I20260926 11:39:58.701370 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab0c4c70c25443539bfad78327c28024. Latest consensus state: current_term: 1 leader_uuid: "ab0c4c70c25443539bfad78327c28024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0c4c70c25443539bfad78327c28024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43561 } } }
I20260926 11:39:58.701502 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024 [sys.catalog]: This master's current role is: LEADER
W20260926 11:39:58.705060 21987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:39:58.705160 21987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:39:58.705262 21983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:39:58.706377 21983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:39:58.713225 21983 catalog_manager.cc:1347] Generated new cluster ID: 3fbad45c8f2945bab8240e823b39ebfd
I20260926 11:39:58.713342 21983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:39:58.739538 21983 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:39:58.739933 21983 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:39:58.746143 21983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab0c4c70c25443539bfad78327c28024: Generated new TSK 0
I20260926 11:39:58.746966 21983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:39:58.752813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 21894
2026-09-26T11:39:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:39:58 dist-test-slave-4pw5 krb5kdc[21993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:39:58 dist-test-slave-4pw5 krb5kdc[21993](info): set up 2 sockets
Sep 26 11:39:58 dist-test-slave-4pw5 krb5kdc[21993](info): commencing operation
krb5kdc: starting...
W20260926 11:39:59.864218 392 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:39:59 dist-test-slave-4pw5 krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:39:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:40:00.000196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43323
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:40:00.130486 22006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:00.130873 22006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:00.130978 22006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:00.136042 22006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:00.136165 22006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:00.136206 22006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:00.136281 22006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:00.136343 22006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:00.142746 22006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:00.144089 22006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:00.145255 22006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:00.155359 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:00.155340 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:00.155341 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:00.156025 22006 server_base.cc:1034] running on GCE node
I20260926 11:40:00.156858 22006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:00.158268 22006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:00.159504 22006 hybrid_clock.cc:648] HybridClock initialized: now 1790422800159486 us; error 39 us; skew 500 ppm
Sep 26 11:40:00 dist-test-slave-4pw5 krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:00.163043 22006 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:40:00.164355 22006 webserver.cc:460] Webserver started at http://127.0.98.62:37907/ using document root <none> and password file <none>
I20260926 11:40:00.165012 22006 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:00.165119 22006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:00.165385 22006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:00.167891 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "388bb51eac634492b33c012c6116f37b"
format_stamp: "Formatted at 2026-09-26 11:40:00 on dist-test-slave-4pw5"
server_key: "dcd2d73db4318d316124da3c8ece473d"
server_key_iv: "1506f83e9d6f536f2b11a48f210a02d9"
server_key_version: "encryptionkey@0"
I20260926 11:40:00.168685 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "388bb51eac634492b33c012c6116f37b"
format_stamp: "Formatted at 2026-09-26 11:40:00 on dist-test-slave-4pw5"
server_key: "dcd2d73db4318d316124da3c8ece473d"
server_key_iv: "1506f83e9d6f536f2b11a48f210a02d9"
server_key_version: "encryptionkey@0"
I20260926 11:40:00.173218 22006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:40:00.176267 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:00.177870 22006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:40:00.178089 22006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "388bb51eac634492b33c012c6116f37b"
format_stamp: "Formatted at 2026-09-26 11:40:00 on dist-test-slave-4pw5"
server_key: "dcd2d73db4318d316124da3c8ece473d"
server_key_iv: "1506f83e9d6f536f2b11a48f210a02d9"
server_key_version: "encryptionkey@0"
I20260926 11:40:00.178251 22006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:00.193049 22006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:00.197243 22006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:00.197541 22006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:00.206240 22006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43323
I20260926 11:40:00.206247 22073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43323 every 8 connection(s)
I20260926 11:40:00.207391 22006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:40:00.208292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 22006
I20260926 11:40:00.208455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:40:00.208737 392 external_mini_cluster.cc:1428] Setting key f6f8fd179e1ba71b4b0ef016a4e46d17
I20260926 11:40:00.211391 22074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:00.217911 22074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b: Bootstrap starting.
Sep 26 11:40:00 dist-test-slave-4pw5 krb5kdc[21993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:00.220976 22074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:00.221953 22074 log.cc:824] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b: Log is configured to *not* fsync() on all Append() calls
I20260926 11:40:00.224392 22074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b: No bootstrap required, opened a new log
I20260926 11:40:00.226300 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:00.210334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37887 (local address 127.0.98.62:43323)
0926 11:40:00.210934 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:00.210952 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:00.210994 (+ 42us) server_negotiation.cc:407] Connection header received
0926 11:40:00.212022 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:00.212044 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:00.212512 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:00.212937 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:00.213868 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:00.214903 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:00.215902 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:00.216584 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:00.216939 (+ 355us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:00.220182 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:00.220188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:00.220203 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:00.220237 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:00.222870 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:00.223565 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:00.223633 (+ 68us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:00.223640 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:00.223781 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:00.224161 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:00.224164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:00.224166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:00.224664 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:00.224922 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:00.225238 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 11:40:00.225491 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":164,"threads_started":1}
I20260926 11:40:00.229637 22074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388bb51eac634492b33c012c6116f37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } }
I20260926 11:40:00.230039 22074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:40:00.230118 22074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388bb51eac634492b33c012c6116f37b, State: Initialized, Role: FOLLOWER
I20260926 11:40:00.230230 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:40:00.230657 22074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388bb51eac634492b33c012c6116f37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } }
I20260926 11:40:00.230795 22074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:40:00.230898 22074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:40:00.231036 22074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:00.233634 22074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388bb51eac634492b33c012c6116f37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } }
I20260926 11:40:00.234174 22074 leader_election.cc:302] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 388bb51eac634492b33c012c6116f37b; no voters:
I20260926 11:40:00.234629 22074 leader_election.cc:288] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:40:00.234712 22084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:40:00.234946 22084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [term 1 LEADER]: Becoming Leader. State: Replica: 388bb51eac634492b33c012c6116f37b, State: Running, Role: LEADER
I20260926 11:40:00.235357 22084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388bb51eac634492b33c012c6116f37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } }
I20260926 11:40:00.236254 22074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:00.237308 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "388bb51eac634492b33c012c6116f37b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388bb51eac634492b33c012c6116f37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } } }
I20260926 11:40:00.237856 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:00.238400 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 388bb51eac634492b33c012c6116f37b. Latest consensus state: current_term: 1 leader_uuid: "388bb51eac634492b33c012c6116f37b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388bb51eac634492b33c012c6116f37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43323 } } }
I20260926 11:40:00.239445 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:00.242995 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:00.233331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39579 (local address 127.0.98.62:43323)
0926 11:40:00.233473 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:00.233477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:00.233638 (+ 161us) server_negotiation.cc:407] Connection header received
0926 11:40:00.233827 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:00.233830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:00.233892 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:00.233989 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:00.234832 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:00.235547 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:00.236507 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:00.236859 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:00.237416 (+ 557us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:00.238882 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:00.238886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:00.238889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:00.238921 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:00.240677 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:00.241287 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:00.241291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:00.241293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:00.241521 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:00.241977 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:00.241980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:00.241982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:00.242236 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:00.242389 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:00.242706 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 11:40:00.242818 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 11:40:00.243233 22099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:40:00.243314 22099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:40:00.243417 22093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:40:00.244215 22093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:40:00.251299 22093 catalog_manager.cc:1347] Generated new cluster ID: 0af217aae6e745709993cdd35c6296e2
I20260926 11:40:00.251402 22093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:40:00.261663 22093 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:40:00.262806 22093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:40:00.269631 22093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 388bb51eac634492b33c012c6116f37b: Generated new TSK 0
I20260926 11:40:00.270560 22093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:40:00.279862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 22006
W20260926 11:40:00.286432 21991 connection.cc:535] client connection to 127.0.98.62:43323 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43323): Connection reset by peer (error 104)
2026-09-26T11:40:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1528 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:40:00 dist-test-slave-4pw5 krb5kdc[22105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:40:00 dist-test-slave-4pw5 krb5kdc[22105](info): set up 2 sockets
Sep 26 11:40:00 dist-test-slave-4pw5 krb5kdc[22105](info): commencing operation
krb5kdc: starting...
W20260926 11:40:01.348522 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:40:01 dist-test-slave-4pw5 krb5kdc[22105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:40:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:40:01.521915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35143
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:40:01.670398 22118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:01.670855 22118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:01.670908 22118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:01.675606 22118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:01.675726 22118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:01.675747 22118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:01.675766 22118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:01.675783 22118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:01.682437 22118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:01.684021 22118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:01.685648 22118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:01.697860 22123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:01.697860 22124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:01.697870 22126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:01.698189 22118 server_base.cc:1034] running on GCE node
I20260926 11:40:01.699025 22118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:01.700608 22118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:01.701854 22118 hybrid_clock.cc:648] HybridClock initialized: now 1790422801701834 us; error 41 us; skew 500 ppm
Sep 26 11:40:01 dist-test-slave-4pw5 krb5kdc[22105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:01.706146 22118 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:40:01.707706 22118 webserver.cc:460] Webserver started at http://127.0.98.62:44995/ using document root <none> and password file <none>
I20260926 11:40:01.708567 22118 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:01.708663 22118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:01.708984 22118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:01.712116 22118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "493059ce0ee544538efa5f85a8a92c01"
format_stamp: "Formatted at 2026-09-26 11:40:01 on dist-test-slave-4pw5"
server_key: "ebc46d741ca0e841e0164988fc1c3239"
server_key_iv: "7f75eadd378567743bbecb5612eb06f5"
server_key_version: "encryptionkey@0"
I20260926 11:40:01.712886 22118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "493059ce0ee544538efa5f85a8a92c01"
format_stamp: "Formatted at 2026-09-26 11:40:01 on dist-test-slave-4pw5"
server_key: "ebc46d741ca0e841e0164988fc1c3239"
server_key_iv: "7f75eadd378567743bbecb5612eb06f5"
server_key_version: "encryptionkey@0"
I20260926 11:40:01.718319 22118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 11:40:01.721983 22133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:01.723824 22118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:40:01.724012 22118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "493059ce0ee544538efa5f85a8a92c01"
format_stamp: "Formatted at 2026-09-26 11:40:01 on dist-test-slave-4pw5"
server_key: "ebc46d741ca0e841e0164988fc1c3239"
server_key_iv: "7f75eadd378567743bbecb5612eb06f5"
server_key_version: "encryptionkey@0"
I20260926 11:40:01.724261 22118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:01.756996 22118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:01.761658 22118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:01.761951 22118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:01.771677 22118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35143
I20260926 11:40:01.771704 22185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35143 every 8 connection(s)
I20260926 11:40:01.773130 22118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:40:01.777432 22186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:01.783223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 22118
I20260926 11:40:01.783381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:40:01.783766 392 external_mini_cluster.cc:1428] Setting key c1ee475e368ac26bca3c63a2d6361813
I20260926 11:40:01.785599 22186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01: Bootstrap starting.
I20260926 11:40:01.788813 22186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:01.789840 22186 log.cc:824] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01: Log is configured to *not* fsync() on all Append() calls
I20260926 11:40:01.792550 22186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01: No bootstrap required, opened a new log
I20260926 11:40:01.796132 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:01.785601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41877 (local address 127.0.98.62:35143)
0926 11:40:01.786208 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:01.786226 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:01.786272 (+ 46us) server_negotiation.cc:407] Connection header received
0926 11:40:01.787118 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:01.787142 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:01.787754 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:01.788211 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:01.788761 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:01.790096 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:01.791351 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:01.792086 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:01.792286 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:01.793589 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:01.793595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:01.793615 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:40:01.794230 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:01.794320 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:40:01.794350 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:01.794549 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:01.794773 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 11:40:01.794983 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":218,"threads_started":1}
I20260926 11:40:01.797726 22186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493059ce0ee544538efa5f85a8a92c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } }
I20260926 11:40:01.798094 22186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:40:01.798185 22186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493059ce0ee544538efa5f85a8a92c01, State: Initialized, Role: FOLLOWER
I20260926 11:40:01.798836 22186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493059ce0ee544538efa5f85a8a92c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } }
I20260926 11:40:01.799054 22186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:40:01.799162 22186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:40:01.799285 22186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:01.800729 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:40:01.800904 22186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493059ce0ee544538efa5f85a8a92c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } }
I20260926 11:40:01.801378 22186 leader_election.cc:302] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 493059ce0ee544538efa5f85a8a92c01; no voters:
I20260926 11:40:01.801862 22186 leader_election.cc:288] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:40:01.801987 22191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:40:01.802340 22191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [term 1 LEADER]: Becoming Leader. State: Replica: 493059ce0ee544538efa5f85a8a92c01, State: Running, Role: LEADER
I20260926 11:40:01.802850 22191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493059ce0ee544538efa5f85a8a92c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } }
I20260926 11:40:01.803767 22186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:01.805368 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "493059ce0ee544538efa5f85a8a92c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493059ce0ee544538efa5f85a8a92c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } } }
I20260926 11:40:01.805524 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:01.806016 22193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493059ce0ee544538efa5f85a8a92c01. Latest consensus state: current_term: 1 leader_uuid: "493059ce0ee544538efa5f85a8a92c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493059ce0ee544538efa5f85a8a92c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } } }
I20260926 11:40:01.806134 22193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01 [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:01.806794 22207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:40:01.809556 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:01.803639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53275 (local address 127.0.98.62:35143)
0926 11:40:01.803801 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:01.803806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:01.804144 (+ 338us) server_negotiation.cc:407] Connection header received
0926 11:40:01.804282 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:01.804286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:01.804348 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:01.804452 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:01.804869 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:01.805811 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:01.807278 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:01.807681 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:01.807833 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:01.808967 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:01.808971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:01.808974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:40:01.808994 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:01.809031 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:40:01.809045 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:01.809150 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:01.809335 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 11:40:01.809381 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 11:40:01.811710 22207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:40:01.820508 22207 catalog_manager.cc:1347] Generated new cluster ID: b08d8f541c3643ecbfaea551849bf537
I20260926 11:40:01.820643 22207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:40:01.832118 22207 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:40:01.833499 22207 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:40:01.848225 22207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 493059ce0ee544538efa5f85a8a92c01: Generated new TSK 0
I20260926 11:40:01.849334 22207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:40:01.858232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 22118
W20260926 11:40:01.864488 22103 connection.cc:535] client connection to 127.0.98.62:35143 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35143): Connection reset by peer (error 104)
2026-09-26T11:40:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1577 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:40:01 dist-test-slave-4pw5 krb5kdc[22217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:40:01 dist-test-slave-4pw5 krb5kdc[22217](info): set up 2 sockets
Sep 26 11:40:01 dist-test-slave-4pw5 krb5kdc[22217](info): commencing operation
krb5kdc: starting...
W20260926 11:40:02.956372 392 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:40:03 dist-test-slave-4pw5 krb5kdc[22217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:40:03.104171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33453
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:40:03.238235 22230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:03.238606 22230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:03.238694 22230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:03.242993 22230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:03.243134 22230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:03.243175 22230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:03.243213 22230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:03.243253 22230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:03.249295 22230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:03.250590 22230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:03.251817 22230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:03.261993 22236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:03.262054 22235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:03.262215 22230 server_base.cc:1034] running on GCE node
W20260926 11:40:03.262054 22238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:03.263263 22230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:03.264923 22230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:03.266155 22230 hybrid_clock.cc:648] HybridClock initialized: now 1790422803266114 us; error 68 us; skew 500 ppm
Sep 26 11:40:03 dist-test-slave-4pw5 krb5kdc[22217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:03.275208 22230 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:40:03.276603 22230 webserver.cc:460] Webserver started at http://127.0.98.62:41563/ using document root <none> and password file <none>
I20260926 11:40:03.277347 22230 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:03.277451 22230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:03.277735 22230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:03.280442 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "158271d57b4b4e0cbc64ffeb323c565a"
format_stamp: "Formatted at 2026-09-26 11:40:03 on dist-test-slave-4pw5"
server_key: "743b35ac89803d5c2f2c5daf62751623"
server_key_iv: "c5d0925a375559847e7cd1a9de0265a8"
server_key_version: "encryptionkey@0"
I20260926 11:40:03.281145 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "158271d57b4b4e0cbc64ffeb323c565a"
format_stamp: "Formatted at 2026-09-26 11:40:03 on dist-test-slave-4pw5"
server_key: "743b35ac89803d5c2f2c5daf62751623"
server_key_iv: "c5d0925a375559847e7cd1a9de0265a8"
server_key_version: "encryptionkey@0"
I20260926 11:40:03.286448 22230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 11:40:03.290167 22245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:03.292186 22230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 11:40:03.292400 22230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "158271d57b4b4e0cbc64ffeb323c565a"
format_stamp: "Formatted at 2026-09-26 11:40:03 on dist-test-slave-4pw5"
server_key: "743b35ac89803d5c2f2c5daf62751623"
server_key_iv: "c5d0925a375559847e7cd1a9de0265a8"
server_key_version: "encryptionkey@0"
I20260926 11:40:03.292583 22230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:03.298558 22230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:03.303051 22230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:03.303337 22230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:03.312935 22230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33453
I20260926 11:40:03.313000 22297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33453 every 8 connection(s)
I20260926 11:40:03.314205 22230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:40:03.317649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 22230
I20260926 11:40:03.317826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:40:03.318132 392 external_mini_cluster.cc:1428] Setting key 5e111f86a3aa177605067785485f3c09
I20260926 11:40:03.318137 22298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:03.325805 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:03.320095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56443 (local address 127.0.98.62:33453)
0926 11:40:03.320692 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:03.320713 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:03.320765 (+ 52us) server_negotiation.cc:407] Connection header received
0926 11:40:03.321939 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:03.321963 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:03.322386 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:03.322810 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:03.324070 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:03.324076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:03.324092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:40:03.324107 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:03.324182 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:40:03.324213 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:03.324384 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:03.324520 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 11:40:03.324782 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":185,"threads_started":1}
I20260926 11:40:03.327642 22298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a: Bootstrap starting.
I20260926 11:40:03.330080 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:40:03.330521 22298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:03.331362 22298 log.cc:824] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:40:03.333920 22298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a: No bootstrap required, opened a new log
I20260926 11:40:03.335484 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:03.333048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53119 (local address 127.0.98.62:33453)
0926 11:40:03.333235 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:03.333242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:03.333557 (+ 315us) server_negotiation.cc:407] Connection header received
0926 11:40:03.333725 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:03.333728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:03.333775 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:03.333864 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:03.334917 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:03.334922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:03.334926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:40:03.334939 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:03.334974 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:40:03.334987 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:03.335111 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:03.335225 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 11:40:03.335272 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 11:40:03.338340 22298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158271d57b4b4e0cbc64ffeb323c565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260926 11:40:03.338676 22298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:40:03.338748 22298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158271d57b4b4e0cbc64ffeb323c565a, State: Initialized, Role: FOLLOWER
I20260926 11:40:03.339294 22298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158271d57b4b4e0cbc64ffeb323c565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260926 11:40:03.339474 22298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:40:03.339569 22298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:40:03.339723 22298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:03.340798 22298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158271d57b4b4e0cbc64ffeb323c565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260926 11:40:03.341261 22298 leader_election.cc:302] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 158271d57b4b4e0cbc64ffeb323c565a; no voters:
I20260926 11:40:03.341667 22298 leader_election.cc:288] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:40:03.341869 22308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:40:03.342259 22308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [term 1 LEADER]: Becoming Leader. State: Replica: 158271d57b4b4e0cbc64ffeb323c565a, State: Running, Role: LEADER
I20260926 11:40:03.342717 22308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158271d57b4b4e0cbc64ffeb323c565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260926 11:40:03.343308 22298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:03.345052 22309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "158271d57b4b4e0cbc64ffeb323c565a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158271d57b4b4e0cbc64ffeb323c565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } } }
I20260926 11:40:03.345249 22309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:03.345602 22310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158271d57b4b4e0cbc64ffeb323c565a. Latest consensus state: current_term: 1 leader_uuid: "158271d57b4b4e0cbc64ffeb323c565a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158271d57b4b4e0cbc64ffeb323c565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } } }
I20260926 11:40:03.345755 22310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:03.345810 22319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:40:03.349539 22319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:40:03.357897 22319 catalog_manager.cc:1347] Generated new cluster ID: 5fe10eef37094a5da1904906c53c9cb5
I20260926 11:40:03.358008 22319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:40:03.391558 22319 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:40:03.392011 22319 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:40:03.397895 22319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 158271d57b4b4e0cbc64ffeb323c565a: Generated new TSK 0
I20260926 11:40:03.399065 22319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:40:03.409788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 22230
2026-09-26T11:40:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1552 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:40:03 dist-test-slave-4pw5 krb5kdc[22329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:40:03 dist-test-slave-4pw5 krb5kdc[22329](info): set up 2 sockets
Sep 26 11:40:03 dist-test-slave-4pw5 krb5kdc[22329](info): commencing operation
krb5kdc: starting...
W20260926 11:40:04.488250 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:40:04 dist-test-slave-4pw5 krb5kdc[22329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:40:04.631705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43375
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:40:04.754012 22342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:04.754339 22342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:04.754387 22342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:04.758651 22342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:04.758730 22342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:04.758750 22342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:04.758767 22342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:04.758783 22342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:04.764405 22342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:04.765581 22342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:04.766693 22342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:04.776556 22348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:04.776574 22347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:04.776721 22342 server_base.cc:1034] running on GCE node
W20260926 11:40:04.776556 22350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:04.777724 22342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:04.779242 22342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:04.780481 22342 hybrid_clock.cc:648] HybridClock initialized: now 1790422804780472 us; error 69 us; skew 500 ppm
Sep 26 11:40:04 dist-test-slave-4pw5 krb5kdc[22329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:04.784149 22342 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:40:04.785570 22342 webserver.cc:460] Webserver started at http://127.0.98.62:45619/ using document root <none> and password file <none>
I20260926 11:40:04.786386 22342 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:04.786494 22342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:04.786769 22342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:04.789572 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "8be57c16135543789c718515f3b7cec0"
format_stamp: "Formatted at 2026-09-26 11:40:04 on dist-test-slave-4pw5"
server_key: "504c056dc01e819a8e87022bb17a5975"
server_key_iv: "5348253ac7d2e2cd17d726823e934595"
server_key_version: "encryptionkey@0"
I20260926 11:40:04.790305 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "8be57c16135543789c718515f3b7cec0"
format_stamp: "Formatted at 2026-09-26 11:40:04 on dist-test-slave-4pw5"
server_key: "504c056dc01e819a8e87022bb17a5975"
server_key_iv: "5348253ac7d2e2cd17d726823e934595"
server_key_version: "encryptionkey@0"
I20260926 11:40:04.795435 22342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 11:40:04.798981 22357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:04.800825 22342 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:40:04.801028 22342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "8be57c16135543789c718515f3b7cec0"
format_stamp: "Formatted at 2026-09-26 11:40:04 on dist-test-slave-4pw5"
server_key: "504c056dc01e819a8e87022bb17a5975"
server_key_iv: "5348253ac7d2e2cd17d726823e934595"
server_key_version: "encryptionkey@0"
I20260926 11:40:04.801193 22342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:04.844384 22342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:04.848615 22342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:04.848857 22342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:04.858253 22342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43375
I20260926 11:40:04.858291 22409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43375 every 8 connection(s)
I20260926 11:40:04.859474 22342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:40:04.860895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 22342
I20260926 11:40:04.861020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:40:04.861413 392 external_mini_cluster.cc:1428] Setting key 7a662f47ea34abb0a4ad28019b50735f
I20260926 11:40:04.863340 22410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:04.870995 22410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0: Bootstrap starting.
Sep 26 11:40:04 dist-test-slave-4pw5 krb5kdc[22329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:04.873983 22410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:04.874792 22410 log.cc:824] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0: Log is configured to *not* fsync() on all Append() calls
I20260926 11:40:04.877543 22410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0: No bootstrap required, opened a new log
I20260926 11:40:04.879525 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:04.863360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54431 (local address 127.0.98.62:43375)
0926 11:40:04.864024 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:04.864058 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:04.864110 (+ 52us) server_negotiation.cc:407] Connection header received
0926 11:40:04.864901 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:04.864922 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:04.865379 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:04.865752 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:04.866976 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:04.868309 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:04.869661 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:04.870341 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:04.870504 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:04.873983 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:04.873991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:04.874012 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:04.874064 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:04.876243 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:04.876807 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:04.876812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:04.876818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:04.876950 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:04.877240 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:04.877245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:04.877247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:04.877780 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:04.877994 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:04.878326 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 11:40:04.878687 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":293,"threads_started":1}
I20260926 11:40:04.882825 22410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be57c16135543789c718515f3b7cec0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43375 } }
I20260926 11:40:04.883231 22410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:40:04.883287 22410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be57c16135543789c718515f3b7cec0, State: Initialized, Role: FOLLOWER
I20260926 11:40:04.883725 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:40:04.883944 22410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be57c16135543789c718515f3b7cec0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43375 } }
I20260926 11:40:04.884147 22410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:40:04.884234 22410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:40:04.884384 22410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:04.885701 22410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be57c16135543789c718515f3b7cec0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43375 } }
I20260926 11:40:04.886173 22410 leader_election.cc:302] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8be57c16135543789c718515f3b7cec0; no voters:
I20260926 11:40:04.886670 22410 leader_election.cc:288] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:40:04.886817 22420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:40:04.887151 22420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [term 1 LEADER]: Becoming Leader. State: Replica: 8be57c16135543789c718515f3b7cec0, State: Running, Role: LEADER
I20260926 11:40:04.887748 22420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be57c16135543789c718515f3b7cec0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43375 } }
I20260926 11:40:04.888584 22410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:04.889986 22422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8be57c16135543789c718515f3b7cec0. Latest consensus state: current_term: 1 leader_uuid: "8be57c16135543789c718515f3b7cec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be57c16135543789c718515f3b7cec0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43375 } } }
I20260926 11:40:04.890169 22422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:04.890573 22421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8be57c16135543789c718515f3b7cec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be57c16135543789c718515f3b7cec0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43375 } } }
I20260926 11:40:04.890686 22421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0 [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:04.891101 22429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:40:04.895483 22429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:40:04.898097 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:04.886420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45843 (local address 127.0.98.62:43375)
0926 11:40:04.886607 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:04.886614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:04.886748 (+ 134us) server_negotiation.cc:407] Connection header received
0926 11:40:04.886914 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:04.886917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:04.886990 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:04.887120 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:04.888261 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:04.889164 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:04.891010 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:04.891429 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:04.891530 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:04.893724 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:04.893728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:04.893731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:04.893763 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:04.895874 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:04.896492 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:04.896497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:04.896499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:04.896619 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:04.896976 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:04.896980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:04.896982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:04.897264 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:04.897415 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:04.897741 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 11:40:04.897903 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:40:04.904645 22429 catalog_manager.cc:1347] Generated new cluster ID: 0e53e71f390d4b3fa9495b17a843355e
I20260926 11:40:04.904767 22429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:40:04.923650 22429 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:40:04.924942 22429 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:40:04.934810 22429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8be57c16135543789c718515f3b7cec0: Generated new TSK 0
I20260926 11:40:04.936077 22429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:40:04.943529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 22342
2026-09-26T11:40:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:40:05 dist-test-slave-4pw5 krb5kdc[22441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:40:05 dist-test-slave-4pw5 krb5kdc[22441](info): set up 2 sockets
Sep 26 11:40:05 dist-test-slave-4pw5 krb5kdc[22441](info): commencing operation
krb5kdc: starting...
W20260926 11:40:06.044265 392 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:40:06 dist-test-slave-4pw5 krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:40:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:40:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:40:06.183200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41141
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:40:06.306702 22454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:06.307024 22454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:06.307070 22454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:06.311528 22454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:06.311707 22454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:06.311733 22454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:06.311795 22454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:06.311820 22454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:06.318059 22454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:06.319319 22454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:06.320484 22454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:06.329958 22462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:06.329958 22460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:06.329973 22459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:06.330235 22454 server_base.cc:1034] running on GCE node
I20260926 11:40:06.330962 22454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:06.332281 22454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:06.333482 22454 hybrid_clock.cc:648] HybridClock initialized: now 1790422806333443 us; error 43 us; skew 500 ppm
Sep 26 11:40:06 dist-test-slave-4pw5 krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:06.336917 22454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:40:06.337976 22454 webserver.cc:460] Webserver started at http://127.0.98.62:45641/ using document root <none> and password file <none>
I20260926 11:40:06.338637 22454 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:06.338707 22454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:06.338958 22454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:06.341446 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "1d2bb84f59d547daa4fa14ea71e675df"
format_stamp: "Formatted at 2026-09-26 11:40:06 on dist-test-slave-4pw5"
server_key: "7667c59d20df73395e13848fda356c5d"
server_key_iv: "b11543da91b8880ac244354dfe3bf76a"
server_key_version: "encryptionkey@0"
I20260926 11:40:06.342116 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d2bb84f59d547daa4fa14ea71e675df"
format_stamp: "Formatted at 2026-09-26 11:40:06 on dist-test-slave-4pw5"
server_key: "7667c59d20df73395e13848fda356c5d"
server_key_iv: "b11543da91b8880ac244354dfe3bf76a"
server_key_version: "encryptionkey@0"
I20260926 11:40:06.346486 22454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 11:40:06.349303 22469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:06.350924 22454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:40:06.351084 22454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "1d2bb84f59d547daa4fa14ea71e675df"
format_stamp: "Formatted at 2026-09-26 11:40:06 on dist-test-slave-4pw5"
server_key: "7667c59d20df73395e13848fda356c5d"
server_key_iv: "b11543da91b8880ac244354dfe3bf76a"
server_key_version: "encryptionkey@0"
I20260926 11:40:06.351219 22454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:06.370972 22454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:06.374842 22454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:06.375095 22454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:06.382951 22454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41141
I20260926 11:40:06.382963 22521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41141 every 8 connection(s)
I20260926 11:40:06.384096 22454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:40:06.387259 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:06.391263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 22454
I20260926 11:40:06.391414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:40:06.391817 392 external_mini_cluster.cc:1428] Setting key 5c4defb70af559137439aea5f01f4677
I20260926 11:40:06.394563 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df: Bootstrap starting.
I20260926 11:40:06.397641 22522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:06.398705 22522 log.cc:824] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df: Log is configured to *not* fsync() on all Append() calls
I20260926 11:40:06.401367 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df: No bootstrap required, opened a new log
I20260926 11:40:06.402855 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:06.393611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42169 (local address 127.0.98.62:41141)
0926 11:40:06.394108 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:06.394125 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:06.394164 (+ 39us) server_negotiation.cc:407] Connection header received
0926 11:40:06.394912 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:06.394933 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:06.395332 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:06.395781 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:06.396265 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:06.397573 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:06.398943 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:06.399540 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:06.399760 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:06.400830 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:06.400836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:06.400850 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:40:06.401338 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:06.401408 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:40:06.401483 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:06.401634 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:06.401791 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 11:40:06.402045 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":187,"threads_started":1}
I20260926 11:40:06.406754 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:40:06.406489 22522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2bb84f59d547daa4fa14ea71e675df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41141 } }
I20260926 11:40:06.406876 22522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:40:06.406978 22522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2bb84f59d547daa4fa14ea71e675df, State: Initialized, Role: FOLLOWER
I20260926 11:40:06.407514 22522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2bb84f59d547daa4fa14ea71e675df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41141 } }
I20260926 11:40:06.407719 22522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:40:06.407815 22522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:40:06.407931 22522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:06.409106 22522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2bb84f59d547daa4fa14ea71e675df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41141 } }
I20260926 11:40:06.409538 22522 leader_election.cc:302] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d2bb84f59d547daa4fa14ea71e675df; no voters:
I20260926 11:40:06.409930 22522 leader_election.cc:288] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:40:06.409999 22532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:40:06.410188 22532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [term 1 LEADER]: Becoming Leader. State: Replica: 1d2bb84f59d547daa4fa14ea71e675df, State: Running, Role: LEADER
I20260926 11:40:06.410597 22532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2bb84f59d547daa4fa14ea71e675df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41141 } }
I20260926 11:40:06.411455 22522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:06.411954 22533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d2bb84f59d547daa4fa14ea71e675df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2bb84f59d547daa4fa14ea71e675df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41141 } } }
I20260926 11:40:06.412093 22533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:06.413081 22534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2bb84f59d547daa4fa14ea71e675df. Latest consensus state: current_term: 1 leader_uuid: "1d2bb84f59d547daa4fa14ea71e675df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2bb84f59d547daa4fa14ea71e675df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41141 } } }
I20260926 11:40:06.413295 22534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:06.414240 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:06.409288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49731 (local address 127.0.98.62:41141)
0926 11:40:06.409413 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:06.409418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:06.409640 (+ 222us) server_negotiation.cc:407] Connection header received
0926 11:40:06.409788 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:06.409791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:06.409840 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:06.409937 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:06.410211 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:06.410919 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:06.411969 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:06.412300 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:06.412633 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:06.413746 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:06.413749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:06.413752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:40:06.413770 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:06.413843 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:40:06.413861 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:06.413935 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:06.414038 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 11:40:06.414097 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 11:40:06.414340 22536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:40:06.417738 22536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:40:06.424505 22536 catalog_manager.cc:1347] Generated new cluster ID: ca48a7bf597146a8b038ee0fa4c3d4bf
I20260926 11:40:06.424612 22536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:40:06.438339 22536 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:40:06.439381 22536 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:40:06.450100 22536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d2bb84f59d547daa4fa14ea71e675df: Generated new TSK 0
I20260926 11:40:06.450919 22536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:40:06.459515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 22454
2026-09-26T11:40:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1515 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:40:06 dist-test-slave-4pw5 krb5kdc[22553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:40:06 dist-test-slave-4pw5 krb5kdc[22553](info): set up 2 sockets
Sep 26 11:40:06 dist-test-slave-4pw5 krb5kdc[22553](info): commencing operation
krb5kdc: starting...
W20260926 11:40:07.528450 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:40:07 dist-test-slave-4pw5 krb5kdc[22553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:40:07.676685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38573
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:40:07.798367 22566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:07.798759 22566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:07.798851 22566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:07.803212 22566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:07.803349 22566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:07.803386 22566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:07.803429 22566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:07.803462 22566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:07.809473 22566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:07.810720 22566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:07.812107 22566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:07.823455 22572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:07.823465 22574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:07.823658 22566 server_base.cc:1034] running on GCE node
W20260926 11:40:07.823455 22571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:07.824689 22566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:07.826532 22566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:07.827819 22566 hybrid_clock.cc:648] HybridClock initialized: now 1790422807827792 us; error 51 us; skew 500 ppm
Sep 26 11:40:07 dist-test-slave-4pw5 krb5kdc[22553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:07.831063 22566 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:40:07.832639 22566 webserver.cc:460] Webserver started at http://127.0.98.62:33931/ using document root <none> and password file <none>
I20260926 11:40:07.833323 22566 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:07.833433 22566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:07.833675 22566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:07.836056 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "3631547d0dec4c078ec7252e0af9fc37"
format_stamp: "Formatted at 2026-09-26 11:40:07 on dist-test-slave-4pw5"
server_key: "cdfb42e80a93cf114e07f80bf88dc339"
server_key_iv: "263ecc06674a2eff7d8ff196f5ee68f7"
server_key_version: "encryptionkey@0"
I20260926 11:40:07.836829 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "3631547d0dec4c078ec7252e0af9fc37"
format_stamp: "Formatted at 2026-09-26 11:40:07 on dist-test-slave-4pw5"
server_key: "cdfb42e80a93cf114e07f80bf88dc339"
server_key_iv: "263ecc06674a2eff7d8ff196f5ee68f7"
server_key_version: "encryptionkey@0"
I20260926 11:40:07.841410 22566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:40:07.844588 22581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:07.846355 22566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:40:07.846573 22566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "3631547d0dec4c078ec7252e0af9fc37"
format_stamp: "Formatted at 2026-09-26 11:40:07 on dist-test-slave-4pw5"
server_key: "cdfb42e80a93cf114e07f80bf88dc339"
server_key_iv: "263ecc06674a2eff7d8ff196f5ee68f7"
server_key_version: "encryptionkey@0"
I20260926 11:40:07.846756 22566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:07.852721 22566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:07.856369 22566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:07.856598 22566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:07.865010 22566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38573
I20260926 11:40:07.865077 22633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38573 every 8 connection(s)
I20260926 11:40:07.866149 22566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:40:07.870246 22634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:07.875306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 22566
I20260926 11:40:07.875525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:40:07.875885 392 external_mini_cluster.cc:1428] Setting key e7d168c220b9e53b642dd221d2a7e913
I20260926 11:40:07.878068 22634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37: Bootstrap starting.
I20260926 11:40:07.881150 22634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:07.882223 22634 log.cc:824] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37: Log is configured to *not* fsync() on all Append() calls
I20260926 11:40:07.882822 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:07.877526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56487 (local address 127.0.98.62:38573)
0926 11:40:07.878257 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:07.878286 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:07.878348 (+ 62us) server_negotiation.cc:407] Connection header received
0926 11:40:07.879202 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:07.879224 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:07.879674 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:07.880120 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:07.881054 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:07.881059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:07.881072 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:40:07.881083 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:07.881138 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:40:07.881158 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:07.881298 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:07.881384 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 11:40:07.881562 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":196,"threads_started":1}
I20260926 11:40:07.885097 22634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37: No bootstrap required, opened a new log
I20260926 11:40:07.886574 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:40:07.890456 22634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3631547d0dec4c078ec7252e0af9fc37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38573 } }
I20260926 11:40:07.890803 22634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:40:07.890868 22634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631547d0dec4c078ec7252e0af9fc37, State: Initialized, Role: FOLLOWER
I20260926 11:40:07.891403 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:07.889215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39563 (local address 127.0.98.62:38573)
0926 11:40:07.889412 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:07.889418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:07.889711 (+ 293us) server_negotiation.cc:407] Connection header received
0926 11:40:07.889878 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:07.889881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:07.889928 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:07.890009 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:07.890886 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:07.890890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:07.890893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:40:07.890906 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:07.890941 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:40:07.890957 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:07.891061 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:07.891173 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 11:40:07.891224 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 11:40:07.891439 22634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3631547d0dec4c078ec7252e0af9fc37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38573 } }
I20260926 11:40:07.891726 22634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:40:07.891806 22634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:40:07.891974 22634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:07.893190 22634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3631547d0dec4c078ec7252e0af9fc37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38573 } }
I20260926 11:40:07.893540 22634 leader_election.cc:302] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3631547d0dec4c078ec7252e0af9fc37; no voters:
I20260926 11:40:07.893854 22634 leader_election.cc:288] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:40:07.894104 22644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:40:07.894469 22644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [term 1 LEADER]: Becoming Leader. State: Replica: 3631547d0dec4c078ec7252e0af9fc37, State: Running, Role: LEADER
I20260926 11:40:07.894961 22644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3631547d0dec4c078ec7252e0af9fc37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38573 } }
I20260926 11:40:07.895447 22634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:07.897554 22645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3631547d0dec4c078ec7252e0af9fc37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3631547d0dec4c078ec7252e0af9fc37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38573 } } }
I20260926 11:40:07.897759 22645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:07.897662 22646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3631547d0dec4c078ec7252e0af9fc37. Latest consensus state: current_term: 1 leader_uuid: "3631547d0dec4c078ec7252e0af9fc37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3631547d0dec4c078ec7252e0af9fc37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38573 } } }
I20260926 11:40:07.898108 22646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37 [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:07.898284 22653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:40:07.902540 22653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:40:07.910394 22653 catalog_manager.cc:1347] Generated new cluster ID: 4a31449331cb406b81b9d7ea62677174
I20260926 11:40:07.910669 22653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:40:07.928527 22653 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:40:07.929042 22653 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:40:07.937116 22653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3631547d0dec4c078ec7252e0af9fc37: Generated new TSK 0
I20260926 11:40:07.938162 22653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:40:07.948079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 22566
2026-09-26T11:40:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1492 ms)
[----------] 18 tests from AuthTokenIssuingTest (27407 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:40:08 dist-test-slave-4pw5 krb5kdc[22665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:40:08 dist-test-slave-4pw5 krb5kdc[22665](info): set up 2 sockets
Sep 26 11:40:08 dist-test-slave-4pw5 krb5kdc[22665](info): commencing operation
krb5kdc: starting...
W20260926 11:40:09.028154 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:40:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:40:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:40:09.163879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45103
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45103,127.0.98.61:35059,127.0.98.60:46409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:40:09.289917 22678 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:40:09.290241 22678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:09.290266 22678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:09.290359 22678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:09.294715 22678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:09.294806 22678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:09.294826 22678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:09.294843 22678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:09.294864 22678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:09.300686 22678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45103,127.0.98.61:35059,127.0.98.60:46409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:09.301891 22678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:09.302954 22678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:09.312009 22686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:09.312057 22683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:09.312215 22678 server_base.cc:1034] running on GCE node
W20260926 11:40:09.312009 22684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:09.313074 22678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:09.314407 22678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:09.315624 22678 hybrid_clock.cc:648] HybridClock initialized: now 1790422809315582 us; error 60 us; skew 500 ppm
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:09.322759 22678 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:40:09.323947 22678 webserver.cc:460] Webserver started at http://127.0.98.62:43425/ using document root <none> and password file <none>
I20260926 11:40:09.324573 22678 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:09.324671 22678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:09.324918 22678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:09.327122 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "5f44c9a884744b468f8e46490ac05539"
format_stamp: "Formatted at 2026-09-26 11:40:09 on dist-test-slave-4pw5"
server_key: "512926c03625190bbad5c073649a8315"
server_key_iv: "b1338cf33e6221e303de47c053cf4756"
server_key_version: "encryptionkey@0"
I20260926 11:40:09.327800 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f44c9a884744b468f8e46490ac05539"
format_stamp: "Formatted at 2026-09-26 11:40:09 on dist-test-slave-4pw5"
server_key: "512926c03625190bbad5c073649a8315"
server_key_iv: "b1338cf33e6221e303de47c053cf4756"
server_key_version: "encryptionkey@0"
I20260926 11:40:09.331983 22678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:40:09.335055 22693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:09.336606 22678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:40:09.336781 22678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "5f44c9a884744b468f8e46490ac05539"
format_stamp: "Formatted at 2026-09-26 11:40:09 on dist-test-slave-4pw5"
server_key: "512926c03625190bbad5c073649a8315"
server_key_iv: "b1338cf33e6221e303de47c053cf4756"
server_key_version: "encryptionkey@0"
I20260926 11:40:09.336921 22678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:09.376418 22678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:09.379820 22678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:09.380059 22678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:09.387379 22678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45103
I20260926 11:40:09.387765 22745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45103 every 8 connection(s)
I20260926 11:40:09.388470 22678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:40:09.391752 22746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:09.393356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 22678
I20260926 11:40:09.393502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:40:09.393833 392 external_mini_cluster.cc:1428] Setting key 7b030cea1c0f332190ffea594eb0a93f
I20260926 11:40:09.397179 22746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:09.415053 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.400058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45103 (local address 127.0.0.1:46270)
0926 11:40:09.400708 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.400748 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:09.401675 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:09.402391 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:09.402396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:09.402868 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:09.403121 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.403135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.404534 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.404538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.405483 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.405499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.406364 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.406367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.406552 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.407203 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:09.407221 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:09.409034 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:09.411679 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.411686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.411695 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.411965 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.412374 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.412376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.412377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.412644 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.413570 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:09.413576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:09.413715 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 11:40:09.413868 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 11:40:09.414164 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":35,"spinlock_wait_cycles":21248,"thread_start_us":179,"threads_started":1}
I20260926 11:40:09.415058 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.400057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.0.98.62:45103)
0926 11:40:09.400548 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.400568 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.400875 (+ 307us) server_negotiation.cc:407] Connection header received
0926 11:40:09.402005 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.402010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.402192 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.402395 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.403312 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.404389 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.405639 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.406175 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.406445 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.409222 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.409232 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.409250 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.409283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.411481 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.412094 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.412103 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.412107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.412266 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.412756 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.412759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.412762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.413419 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.413581 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.413998 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.414268 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":330,"spinlock_wait_cycles":1280,"thread_start_us":163,"threads_started":1}
I20260926 11:40:09.417695 22746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:09.418566 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.418236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35059 (local address 127.0.0.1:50930)
0926 11:40:09.418380 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.418454 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 11:40:09.418766 22697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
I20260926 11:40:09.419850 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.419560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35059 (local address 127.0.0.1:50936)
0926 11:40:09.419727 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.419786 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 11:40:09.420028 22746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35059: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 11:40:09.425304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35059
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45103,127.0.98.61:35059,127.0.98.60:46409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:40:09.469340 22746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } attempt: 1
I20260926 11:40:09.470445 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.470039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35059 (local address 127.0.0.1:50952)
0926 11:40:09.470234 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.470313 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 11:40:09.471351 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.471093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35059 (local address 127.0.0.1:50966)
0926 11:40:09.471217 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.471279 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 11:40:09.471606 22746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35059: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
I20260926 11:40:09.539819 22746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } attempt: 2
I20260926 11:40:09.540905 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.540565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35059 (local address 127.0.0.1:50974)
0926 11:40:09.540740 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.540812 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 11:40:09.541988 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.541711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35059 (local address 127.0.0.1:50988)
0926 11:40:09.541848 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.541907 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 11:40:09.542128 22746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35059: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35059: connect: Connection refused (error 111)
W20260926 11:40:09.546912 22753 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:40:09.547253 22753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:09.547278 22753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:09.547389 22753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:09.551875 22753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:09.551952 22753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:09.551980 22753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:09.551998 22753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:09.552014 22753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:09.557852 22753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45103,127.0.98.61:35059,127.0.98.60:46409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:09.559092 22753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:09.560241 22753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:09.570129 22758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:09.570142 22761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:09.570431 22753 server_base.cc:1034] running on GCE node
W20260926 11:40:09.570261 22759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:09.571162 22753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:09.572525 22753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:09.573721 22753 hybrid_clock.cc:648] HybridClock initialized: now 1790422809573702 us; error 38 us; skew 500 ppm
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:09.577069 22753 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 11:40:09.578097 22753 webserver.cc:460] Webserver started at http://127.0.98.61:43049/ using document root <none> and password file <none>
I20260926 11:40:09.578717 22753 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:09.578814 22753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:09.579061 22753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:09.581315 22753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/data/instance:
uuid: "ecaa9b50a0ca484684d632425cde90bd"
format_stamp: "Formatted at 2026-09-26 11:40:09 on dist-test-slave-4pw5"
server_key: "0ee349f4e12ce197021c10851f8e464c"
server_key_iv: "53160b932d624757c6bdd9f6986bb8b1"
server_key_version: "encryptionkey@0"
I20260926 11:40:09.581903 22753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/wal/instance:
uuid: "ecaa9b50a0ca484684d632425cde90bd"
format_stamp: "Formatted at 2026-09-26 11:40:09 on dist-test-slave-4pw5"
server_key: "0ee349f4e12ce197021c10851f8e464c"
server_key_iv: "53160b932d624757c6bdd9f6986bb8b1"
server_key_version: "encryptionkey@0"
I20260926 11:40:09.586150 22753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:40:09.589026 22768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:09.590667 22753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:40:09.590893 22753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/wal
uuid: "ecaa9b50a0ca484684d632425cde90bd"
format_stamp: "Formatted at 2026-09-26 11:40:09 on dist-test-slave-4pw5"
server_key: "0ee349f4e12ce197021c10851f8e464c"
server_key_iv: "53160b932d624757c6bdd9f6986bb8b1"
server_key_version: "encryptionkey@0"
I20260926 11:40:09.591051 22753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:09.608395 22753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:09.612094 22753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:09.612334 22753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:09.620137 22753 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35059
I20260926 11:40:09.620458 22820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35059 every 8 connection(s)
I20260926 11:40:09.621423 22753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:40:09.623021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 22753
I20260926 11:40:09.623160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-1/wal/instance
I20260926 11:40:09.623473 392 external_mini_cluster.cc:1428] Setting key 24c963decb06cbbd28363aaf35a46c66
I20260926 11:40:09.625227 22821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:09.629873 22821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 11:40:09.633296 22746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:09.646505 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.633893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35059 (local address 127.0.0.1:50990)
0926 11:40:09.634022 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.634038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:09.634121 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:09.635256 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:09.635260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:09.635503 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:09.635685 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.635694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.637632 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.637635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.638196 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.638204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.639172 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.639178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.639390 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.640058 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:09.640078 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:09.641822 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:09.644376 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.644379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.644380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.644625 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.645045 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.645047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.645048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.645178 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.645862 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:09.645864 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:09.645935 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 11:40:09.646090 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 11:40:09.646284 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":27}
I20260926 11:40:09.646528 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.632102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.0.98.62:45103)
0926 11:40:09.632253 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.632257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.633544 (+ 1287us) server_negotiation.cc:407] Connection header received
0926 11:40:09.634999 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.635005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.635070 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.635131 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.636364 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.637305 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.638944 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.639388 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.639534 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.642531 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.642534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.642537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.642564 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.644271 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.644814 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.644817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.644818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.644887 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.645314 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.645319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.645325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.645556 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.645634 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.646090 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.646341 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":55}
I20260926 11:40:09.647550 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.632669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45103 (local address 127.0.0.1:46272)
0926 11:40:09.633337 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.633405 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:09.634695 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:09.635373 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:09.635379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:09.635914 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:09.636190 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.636207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.637450 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.637454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.638808 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.638823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.639753 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.639758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.639881 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.640488 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:09.640512 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:09.642378 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:09.644385 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.644393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.644404 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.644643 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.644980 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.644984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.644986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.645122 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.645661 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:09.645667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:09.645815 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 11:40:09.646052 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 11:40:09.646361 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":72,"thread_start_us":153,"threads_started":1}
I20260926 11:40:09.647553 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.634068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.0.98.61:35059)
0926 11:40:09.634502 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.634511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.634562 (+ 51us) server_negotiation.cc:407] Connection header received
0926 11:40:09.634765 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.634784 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.635067 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.635324 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.635871 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.637511 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.638332 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.639033 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.639192 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.642057 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.642065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.642081 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.642106 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.644237 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.644746 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.644756 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.644760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.644896 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.645278 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.645281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.645283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.645755 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.645881 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.646804 (+ 923us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.646998 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":302,"thread_start_us":149,"threads_started":1}
I20260926 11:40:09.649435 22821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:09.649708 22746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:09.650439 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.650151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38188)
0926 11:40:09.650291 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.650352 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 11:40:09.651367 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.651034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38200)
0926 11:40:09.651217 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.651275 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260926 11:40:09.651518 22746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46409: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 11:40:09.656270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46409
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45103,127.0.98.61:35059,127.0.98.60:46409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:40:09.658902 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.650141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35059 (local address 127.0.0.1:50996)
0926 11:40:09.650335 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.650352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:09.650456 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:09.650888 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:09.650893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:09.651097 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:09.651218 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.651225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.652180 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.652183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.652759 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.652768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.653307 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.653310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.653384 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.653955 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:09.653966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:09.655311 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:09.657300 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.657307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.657312 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.657598 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.657921 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.657923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.657924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.657995 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.658435 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:09.658437 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:09.658545 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 11:40:09.658628 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 11:40:09.658735 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 11:40:09.659055 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.650233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.0.98.61:35059)
0926 11:40:09.650651 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.650658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.650674 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:40:09.650711 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.650714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.650755 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.650842 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.651344 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.652080 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.652897 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.653206 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.653294 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.655506 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.655513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.655519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.655549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.657182 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.657721 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.657726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.657730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.657827 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.658083 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.658087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.658089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.658344 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.658449 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.658715 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.658839 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260926 11:40:09.659806 22821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:09.660710 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.660381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38214)
0926 11:40:09.660544 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.660646 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 11:40:09.661026 22772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
I20260926 11:40:09.662087 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.661825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38224)
0926 11:40:09.661959 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.662031 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 11:40:09.662349 22821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46409: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
I20260926 11:40:09.682742 22746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } attempt: 1
I20260926 11:40:09.683862 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.683434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38226)
0926 11:40:09.683703 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.683766 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 11:40:09.684643 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.684409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38242)
0926 11:40:09.684526 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.684582 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 11:40:09.684763 22746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46409: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
I20260926 11:40:09.711676 22821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } attempt: 1
I20260926 11:40:09.712816 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.712311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38256)
0926 11:40:09.712665 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.712730 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248}
I20260926 11:40:09.713701 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.713439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38264)
0926 11:40:09.713576 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.713636 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 11:40:09.713851 22821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46409: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
I20260926 11:40:09.759972 22746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } attempt: 2
I20260926 11:40:09.761256 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.760955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38268)
0926 11:40:09.761103 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.761173 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 11:40:09.762034 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.761815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38276)
0926 11:40:09.761924 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.761977 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 11:40:09.762168 22746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46409: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
W20260926 11:40:09.779047 22827 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:40:09.779451 22827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:09.779518 22827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:09.779625 22827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 11:40:09.782039 22821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } attempt: 2
I20260926 11:40:09.783092 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.782658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38284)
0926 11:40:09.782925 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.783010 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 11:40:09.783936 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.783713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38290)
0926 11:40:09.783822 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.783878 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 11:40:09.784106 22821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46409: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46409: connect: Connection refused (error 111)
W20260926 11:40:09.784469 22827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:09.784548 22827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:09.784590 22827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:09.784626 22827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:09.784657 22827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:09.790515 22827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45103,127.0.98.61:35059,127.0.98.60:46409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:09.791801 22827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:09.792860 22827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:09.802280 22834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:09.802232 22833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:09.802385 22827 server_base.cc:1034] running on GCE node
W20260926 11:40:09.802237 22836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:09.803151 22827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:09.804453 22827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:09.805663 22827 hybrid_clock.cc:648] HybridClock initialized: now 1790422809805648 us; error 33 us; skew 500 ppm
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:09.808835 22827 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 11:40:09.809823 22827 webserver.cc:460] Webserver started at http://127.0.98.60:41823/ using document root <none> and password file <none>
I20260926 11:40:09.810447 22827 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:09.810550 22827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:09.810803 22827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:09.813175 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/data/instance:
uuid: "9ae28ec469684df392b87ab452e01b94"
format_stamp: "Formatted at 2026-09-26 11:40:09 on dist-test-slave-4pw5"
server_key: "54080b96b29bad583edb6df0265abd73"
server_key_iv: "f3ec0c6197f0838b56c9622c365d2088"
server_key_version: "encryptionkey@0"
I20260926 11:40:09.813824 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/wal/instance:
uuid: "9ae28ec469684df392b87ab452e01b94"
format_stamp: "Formatted at 2026-09-26 11:40:09 on dist-test-slave-4pw5"
server_key: "54080b96b29bad583edb6df0265abd73"
server_key_iv: "f3ec0c6197f0838b56c9622c365d2088"
server_key_version: "encryptionkey@0"
I20260926 11:40:09.818463 22827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 11:40:09.821477 22843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:09.822916 22827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:40:09.823086 22827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/wal
uuid: "9ae28ec469684df392b87ab452e01b94"
format_stamp: "Formatted at 2026-09-26 11:40:09 on dist-test-slave-4pw5"
server_key: "54080b96b29bad583edb6df0265abd73"
server_key_iv: "f3ec0c6197f0838b56c9622c365d2088"
server_key_version: "encryptionkey@0"
I20260926 11:40:09.823225 22827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:09.837249 22827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:09.841293 22827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:09.841557 22827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:09.849180 22827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46409
I20260926 11:40:09.849507 22895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46409 every 8 connection(s)
I20260926 11:40:09.850234 22827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
I20260926 11:40:09.853510 22896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:09.854344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 22827
I20260926 11:40:09.854526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422511662684-392-0/minicluster-data/master-2/wal/instance
I20260926 11:40:09.854818 392 external_mini_cluster.cc:1428] Setting key 7e2221bc98b1877214f147da0c709759
I20260926 11:40:09.858358 22896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:09.861332 22746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } attempt: 3
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:09.872056 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.856217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.0.98.60:46409)
0926 11:40:09.856661 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.856683 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.856719 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:40:09.857389 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.857407 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.857742 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.858029 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.859093 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.860283 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.861612 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.862452 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.862626 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.865754 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.865763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.865779 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.865817 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.867874 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.868459 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.868467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.868474 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.868645 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.869073 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.869078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.869080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.869737 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.869959 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.870487 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.870816 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":163,"threads_started":1}
I20260926 11:40:09.872282 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.861855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.0.98.60:46409)
0926 11:40:09.862137 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.862146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.862160 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:40:09.862196 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.862199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.862243 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.862361 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.862822 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.863562 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.864686 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.865079 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.865618 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.868091 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.868094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.868096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.868119 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.870355 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.870882 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.870884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.870886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.870968 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.871273 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.871276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.871277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.871512 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.871643 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.871954 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.872075 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":219,"thread_start_us":90,"threads_started":1}
I20260926 11:40:09.872359 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.860677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.0.98.62:45103)
0926 11:40:09.860844 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.860849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.861136 (+ 287us) server_negotiation.cc:407] Connection header received
0926 11:40:09.861267 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.861270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.861325 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.861429 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.862499 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.863261 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.864783 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.865033 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.865176 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.868499 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.868504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.868509 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.868540 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.870289 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.870877 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.870881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.870883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.870978 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.871316 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.871320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.871322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.871640 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.871737 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.872085 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.872209 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:40:09.872895 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.860492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45103 (local address 127.0.0.1:46280)
0926 11:40:09.860998 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.861049 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:09.861169 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:09.861624 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:09.861633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:09.862195 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:09.862366 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.862375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.863381 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.863386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.864611 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.864623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.865210 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.865214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.865334 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.865864 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:09.865875 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:09.868362 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:09.870406 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.870415 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.870421 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.870789 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.871103 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.871110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.871113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.871204 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.871728 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:09.871734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:09.871845 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 11:40:09.872549 (+ 704us) client_negotiation.cc:240] Negotiation successful
0926 11:40:09.872728 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":204,"threads_started":1}
I20260926 11:40:09.873049 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.861770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38310)
0926 11:40:09.861934 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.861948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:09.862023 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:09.862406 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:09.862413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:09.862613 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:09.862697 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.862703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.863698 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.863702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.864551 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.864562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.865302 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.865307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.865418 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.866068 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:09.866084 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:09.867874 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:09.870487 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.870491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.870493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.870740 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.871065 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.871070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.871075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.871173 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.871649 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:09.871652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:09.871772 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 11:40:09.871879 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 11:40:09.872042 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 11:40:09.874025 22896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:09.875252 22821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } attempt: 3
I20260926 11:40:09.877964 22746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539: Bootstrap starting.
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:09.881282 22746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:09.882550 22746 log.cc:824] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:40:09.885286 22746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539: No bootstrap required, opened a new log
I20260926 11:40:09.885574 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.874681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.0.98.61:35059)
0926 11:40:09.874830 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.874835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.874849 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:40:09.874905 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.874908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.874957 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.875046 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.875664 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.876257 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.877033 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.877276 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.877384 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.879973 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.879977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.879979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.880006 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.882412 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.883520 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.883524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.883526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.883657 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.884011 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.884014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.884016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.884270 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.884341 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.884847 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.884965 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:40:09.886155 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.874897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.0.98.62:45103)
0926 11:40:09.875201 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.875205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.875216 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:40:09.875250 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.875253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.875290 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.875360 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.876720 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.877417 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.878447 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.878748 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.878846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.881689 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.881692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.881695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.881724 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.883725 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.884296 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.884300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.884302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.884396 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.884727 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.884731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.884732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.884974 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.885156 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.885848 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.885987 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260926 11:40:09.886785 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.874558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35059 (local address 127.0.0.1:51002)
0926 11:40:09.874677 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.874692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:09.874786 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:09.875181 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:09.875184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:09.875391 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:09.875511 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.875518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.876366 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.876372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.876900 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.876907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.877386 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.877389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.877496 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.878017 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:09.878037 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:09.879835 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:09.882546 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.882549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.882553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.883391 (+ 838us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.883785 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.883790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.883795 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.883888 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.884526 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:09.884529 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:09.884628 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 11:40:09.886390 (+ 1762us) client_negotiation.cc:240] Negotiation successful
0926 11:40:09.886588 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 11:40:09.887738 22896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:09.889147 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.875648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38322)
0926 11:40:09.875750 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.875764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:09.875827 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:09.879262 (+ 3435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:09.879267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:09.879470 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:09.879606 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.879614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.880764 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.880773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.881740 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.881754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.882660 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.882667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.882779 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.883187 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:09.883202 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:09.884718 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:09.886900 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.886904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.886905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.887159 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.887735 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.887740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.887742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.887861 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.888641 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:09.888645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:09.888744 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 11:40:09.888827 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 11:40:09.888973 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 11:40:09.889659 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.875875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.0.98.60:46409)
0926 11:40:09.879007 (+ 3132us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.879011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.879023 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:40:09.879056 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.879058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.879097 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.879214 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.879739 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.880609 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.882109 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.882530 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.883000 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.884858 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.884861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.884864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.884884 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.886774 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.887496 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.887500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.887502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.887614 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.888285 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.888290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.888292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.888526 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.889321 (+ 795us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.889393 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.889501 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3059}
I20260926 11:40:09.890615 22746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } }
I20260926 11:40:09.890902 22746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f44c9a884744b468f8e46490ac05539, State: Initialized, Role: FOLLOWER
I20260926 11:40:09.891294 22746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } }
I20260926 11:40:09.893204 22746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:09.893293 22821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd: Bootstrap starting.
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:40:09.896577 22821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd: Neither blocks nor log segments found. Creating new log.
Sep 26 11:40:09 dist-test-slave-4pw5 krb5kdc[22665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:40:09.897601 22821 log.cc:824] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd: Log is configured to *not* fsync() on all Append() calls
I20260926 11:40:09.898157 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.898300 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:09.900198 22821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd: No bootstrap required, opened a new log
I20260926 11:40:09.900326 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.888310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46409 (local address 127.0.0.1:38338)
0926 11:40:09.888966 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:09.888979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:09.889054 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:09.889610 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:09.889613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:09.889782 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:09.889877 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.889884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.891048 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.891052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.891820 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:09.891833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.892747 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.892751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:09.892840 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.893424 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:09.893439 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:09.895029 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:09.898561 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.898566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.898568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.898795 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.899148 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:09.899151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:09.899153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:09.899236 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:09.899744 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:09.899747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:09.899852 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 11:40:09.899969 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 11:40:09.900126 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580}
I20260926 11:40:09.900961 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.888524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.0.98.60:46409)
0926 11:40:09.889139 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.889143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.889157 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:40:09.889191 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.889435 (+ 244us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.889488 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.889585 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.890220 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.890925 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.891946 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.892264 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.892362 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.895170 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.895174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.895177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.895200 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.897396 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.898932 (+ 1536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.898936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.898938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.899035 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.899335 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.899338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.899339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.899607 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.899708 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.900657 (+ 949us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.900781 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:40:09.902457 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.888670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.0.98.61:35059)
0926 11:40:09.888783 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.888787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.888803 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:40:09.888868 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.888871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.888916 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.889015 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.890834 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.891657 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.893124 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.893468 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.893554 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.897875 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.897880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.897883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.897915 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.900274 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.900830 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.900834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.900836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.900922 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.901229 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.901232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.901234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.901458 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.901620 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.902170 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.902293 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:40:09.904688 22896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94: Bootstrap starting.
I20260926 11:40:09.900635 22678 master_runner.cc:186] Error getting master registration for 127.0.98.61:35059: OK, instance_id { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" instance_seqno: 1790422809618011 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:40:09.905171 22678 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:40:09.905740 22821 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } }
I20260926 11:40:09.906113 22821 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecaa9b50a0ca484684d632425cde90bd, State: Initialized, Role: FOLLOWER
W20260926 11:40:09.906354 22914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:40:09.906432 22914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:40:09.906630 22821 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } }
I20260926 11:40:09.907781 22896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:09.908411 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:40:09.908560 22821 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:09.908918 22896 log.cc:824] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94: Log is configured to *not* fsync() on all Append() calls
I20260926 11:40:09.909685 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.909840 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:09.909965 22720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5f44c9a884744b468f8e46490ac05539"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46284
I20260926 11:40:09.910228 22720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 11:40:09.910379 22720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:09.911928 22896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94: No bootstrap required, opened a new log
I20260926 11:40:09.912041 22720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } }
I20260926 11:40:09.913744 22795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f44c9a884744b468f8e46490ac05539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ecaa9b50a0ca484684d632425cde90bd"
I20260926 11:40:09.914032 22795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:09.915513 22795 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f44c9a884744b468f8e46490ac05539 in term 1.
I20260926 11:40:09.916110 22697 leader_election.cc:302] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f44c9a884744b468f8e46490ac05539, ecaa9b50a0ca484684d632425cde90bd; no voters:
I20260926 11:40:09.916275 22896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } }
I20260926 11:40:09.916554 22903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:40:09.916567 22896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae28ec469684df392b87ab452e01b94, State: Initialized, Role: FOLLOWER
I20260926 11:40:09.916814 22903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [term 1 LEADER]: Becoming Leader. State: Replica: 5f44c9a884744b468f8e46490ac05539, State: Running, Role: LEADER
I20260926 11:40:09.916914 22896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } }
I20260926 11:40:09.917162 22903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } }
I20260926 11:40:09.917568 22931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.917799 22931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:09.918484 22896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:09.919162 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f44c9a884744b468f8e46490ac05539. Latest consensus state: current_term: 1 leader_uuid: "5f44c9a884744b468f8e46490ac05539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.919291 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:09.923025 22720 leader_election.cc:288] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [CANDIDATE]: Term 1 election: Requested vote from peers ecaa9b50a0ca484684d632425cde90bd (127.0.98.61:35059), 9ae28ec469684df392b87ab452e01b94 (127.0.98.60:46409)
I20260926 11:40:09.924019 22753 master_runner.cc:232] Existing masters have differing terms: 1 vs 0
I20260926 11:40:09.924139 22753 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 11:40:09.925494 22944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:40:09.925590 22944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:40:09.927126 22870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f44c9a884744b468f8e46490ac05539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ae28ec469684df392b87ab452e01b94"
I20260926 11:40:09.927450 22870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:09.928941 22870 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f44c9a884744b468f8e46490ac05539 in term 1.
W20260926 11:40:09.933449 22929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:40:09.933564 22929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:40:09.933923 22933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:40:09.935078 22933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:40:09.940243 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.926002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.0.98.62:45103)
0926 11:40:09.928282 (+ 2280us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.928286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.928300 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:40:09.928338 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.928341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.928399 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.928481 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.930620 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.931432 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.932686 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.933027 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.933099 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.935306 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.935311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.935326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.935356 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.937357 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.938788 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.938792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.938794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.938889 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.939180 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.939184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.939186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.939409 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.939523 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.939892 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.940024 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2188}
I20260926 11:40:09.942137 22870 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [term 1 FOLLOWER]: Refusing update from remote peer 5f44c9a884744b468f8e46490ac05539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:40:09.942914 22903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:40:09.946197 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.926547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.0.98.61:35059)
0926 11:40:09.926683 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.926687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.927228 (+ 541us) server_negotiation.cc:407] Connection header received
0926 11:40:09.927404 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.927410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.927459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.927510 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.931675 (+ 4165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.932394 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.935112 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.935437 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.935524 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.942066 (+ 6542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.942071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.942074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.942108 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.944208 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.944768 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.944772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.944774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.944866 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.945157 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.945160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.945161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.945401 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.945519 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.945879 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.946005 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:40:09.947552 22931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f44c9a884744b468f8e46490ac05539. Latest consensus state: current_term: 1 leader_uuid: "5f44c9a884744b468f8e46490ac05539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.947719 22931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:09.949213 22795 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [term 1 FOLLOWER]: Refusing update from remote peer 5f44c9a884744b468f8e46490ac05539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:40:09.949879 22903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:40:09.951642 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:09.926899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.0.98.60:46409)
0926 11:40:09.929721 (+ 2822us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:09.929726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:09.933791 (+ 4065us) server_negotiation.cc:407] Connection header received
0926 11:40:09.934055 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:09.934059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:09.934113 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:09.934167 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:09.936059 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.936977 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.938153 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:09.938470 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:09.938553 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:09.947707 (+ 9154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:09.947711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:09.947714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:09.947747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:09.949596 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.950154 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.950160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.950163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.950256 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:09.950586 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:09.950589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:09.950591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:09.950811 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:09.950938 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:09.951279 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 11:40:09.951418 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2750}
I20260926 11:40:09.953934 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f44c9a884744b468f8e46490ac05539. Latest consensus state: current_term: 1 leader_uuid: "5f44c9a884744b468f8e46490ac05539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.954092 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:09.962152 22957 mvcc.cc:205] Tried to move back new op lower bound from 7333571829512585216 to 7333571829424975872. Current Snapshot: MvccSnapshot[applied={T|T < 7333571829512585216}]
I20260926 11:40:09.962111 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f44c9a884744b468f8e46490ac05539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.962260 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:09.962357 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f44c9a884744b468f8e46490ac05539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.962442 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539 [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:09.962752 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f44c9a884744b468f8e46490ac05539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.962888 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:09.963922 22954 mvcc.cc:205] Tried to move back new op lower bound from 7333571829512585216 to 7333571829424975872. Current Snapshot: MvccSnapshot[applied={T|T < 7333571829512585216}]
I20260926 11:40:09.964625 22931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f44c9a884744b468f8e46490ac05539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f44c9a884744b468f8e46490ac05539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45103 } } peers { permanent_uuid: "ecaa9b50a0ca484684d632425cde90bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35059 } } peers { permanent_uuid: "9ae28ec469684df392b87ab452e01b94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46409 } } }
I20260926 11:40:09.964758 22931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:09.970465 22933 catalog_manager.cc:1347] Generated new cluster ID: 5c137dbf871d45f89fc895bd72d055ac
I20260926 11:40:09.970563 22933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:40:10.006088 22933 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:40:10.007493 22933 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:40:10.031785 22933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f44c9a884744b468f8e46490ac05539: Generated new TSK 0
I20260926 11:40:10.032958 22933 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:40:10.068195 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.063950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.0.98.61:35059)
0926 11:40:10.064117 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.064123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.066637 (+ 2514us) server_negotiation.cc:407] Connection header received
0926 11:40:10.066871 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.066876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.066932 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.067003 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.068037 (+ 1034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51032: BlockingRecv error: recv got EOF from 127.0.0.1:51032 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 11:40:10.070032 22901 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.065059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.0.98.60:46409)
0926 11:40:10.065187 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.065192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.068948 (+ 3756us) server_negotiation.cc:407] Connection header received
0926 11:40:10.069163 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.069166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.069227 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.069284 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.069875 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38362: BlockingRecv error: recv got EOF from 127.0.0.1:38362 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 11:40:10.072002 22749 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.065891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.0.98.62:45103)
0926 11:40:10.066058 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.066062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.070609 (+ 4547us) server_negotiation.cc:407] Connection header received
0926 11:40:10.071092 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.071096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.071159 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.071211 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.071851 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46310: BlockingRecv error: recv got EOF from 127.0.0.1:46310 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 11:40:10.077627 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.075800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.0.98.61:35059)
0926 11:40:10.075930 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.075934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.076533 (+ 599us) server_negotiation.cc:407] Connection header received
0926 11:40:10.076778 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.076782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.076869 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.076925 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.077512 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51048: BlockingRecv error: recv got EOF from 127.0.0.1:51048 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 11:40:10.083246 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.081476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.0.98.61:35059)
0926 11:40:10.081599 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.081605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.082216 (+ 611us) server_negotiation.cc:407] Connection header received
0926 11:40:10.082419 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.082422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.082481 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.082542 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.083126 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51060: BlockingRecv error: recv got EOF from 127.0.0.1:51060 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 11:40:10.089659 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.087828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.0.98.61:35059)
0926 11:40:10.087952 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.087957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.088581 (+ 624us) server_negotiation.cc:407] Connection header received
0926 11:40:10.088808 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.088812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.088870 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.088928 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.089538 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51066: BlockingRecv error: recv got EOF from 127.0.0.1:51066 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 11:40:10.098393 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.096578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.0.98.61:35059)
0926 11:40:10.096742 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.096747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.097372 (+ 625us) server_negotiation.cc:407] Connection header received
0926 11:40:10.097593 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.097597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.097652 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.097705 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.098269 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51074: BlockingRecv error: recv got EOF from 127.0.0.1:51074 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 11:40:10.111364 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.109425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.0.98.61:35059)
0926 11:40:10.109561 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.109565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.110271 (+ 706us) server_negotiation.cc:407] Connection header received
0926 11:40:10.110510 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.110514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.110584 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.110638 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.111235 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51076: BlockingRecv error: recv got EOF from 127.0.0.1:51076 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 11:40:10.131482 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.130153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.0.98.61:35059)
0926 11:40:10.130285 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.130290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.130548 (+ 258us) server_negotiation.cc:407] Connection header received
0926 11:40:10.130724 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.130728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.130784 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.130886 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.131355 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51082: BlockingRecv error: recv got EOF from 127.0.0.1:51082 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 11:40:10.167111 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.165598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.0.98.61:35059)
0926 11:40:10.165734 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.165739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.165981 (+ 242us) server_negotiation.cc:407] Connection header received
0926 11:40:10.166168 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.166172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.166232 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.166324 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.166985 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51084: BlockingRecv error: recv got EOF from 127.0.0.1:51084 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 11:40:10.234762 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.233568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.0.98.61:35059)
0926 11:40:10.233721 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.233726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.233813 (+ 87us) server_negotiation.cc:407] Connection header received
0926 11:40:10.233969 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.233972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.234040 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.234118 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.234625 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51090: BlockingRecv error: recv got EOF from 127.0.0.1:51090 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 11:40:10.366513 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.365291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.0.98.61:35059)
0926 11:40:10.365447 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.365450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.365495 (+ 45us) server_negotiation.cc:407] Connection header received
0926 11:40:10.365657 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.365660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.365709 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.365784 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.366387 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51098: BlockingRecv error: recv got EOF from 127.0.0.1:51098 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 11:40:10.626189 22825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:10.625066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.0.98.61:35059)
0926 11:40:10.625197 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:10.625202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:10.625300 (+ 98us) server_negotiation.cc:407] Connection header received
0926 11:40:10.625433 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:10.625436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:10.625495 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:10.625588 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:10.626072 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51102: BlockingRecv error: recv got EOF from 127.0.0.1:51102 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 11:40:10.927492 22944 catalog_manager.cc:1259] Loaded cluster ID: 5c137dbf871d45f89fc895bd72d055ac
I20260926 11:40:10.927645 22944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94: loading cluster ID for follower catalog manager: success
I20260926 11:40:10.929647 22944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94: acquiring CA information for follower catalog manager: success
I20260926 11:40:10.931041 22944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ae28ec469684df392b87ab452e01b94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:40:10.935170 22929 catalog_manager.cc:1259] Loaded cluster ID: 5c137dbf871d45f89fc895bd72d055ac
I20260926 11:40:10.935269 22929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd: loading cluster ID for follower catalog manager: success
I20260926 11:40:10.937340 22929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd: acquiring CA information for follower catalog manager: success
I20260926 11:40:10.938890 22929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ecaa9b50a0ca484684d632425cde90bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:40:11.145809 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:11.140943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.0.98.61:35059)
0926 11:40:11.141365 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:11.141371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:11.141400 (+ 29us) server_negotiation.cc:407] Connection header received
0926 11:40:11.141552 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:11.141555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:11.141714 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:11.141839 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:11.142193 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.142755 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.143957 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.144198 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.144291 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:11.144774 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:11.145306 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:11.145483 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:11.145546 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 11:40:11.145609 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":115,"threads_started":1}
I20260926 11:40:11.155122 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:11.150269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.0.98.60:46409)
0926 11:40:11.150638 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:11.150643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:11.150694 (+ 51us) server_negotiation.cc:407] Connection header received
0926 11:40:11.150960 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:11.150963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:11.151125 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:11.151248 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:11.151564 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.152176 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.153280 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.153526 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.153621 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:11.154031 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:11.154494 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:11.154594 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:11.154735 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 11:40:11.154794 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260926 11:40:11.164481 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:11.159149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.0.98.62:45103)
0926 11:40:11.159441 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:11.159447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:11.159850 (+ 403us) server_negotiation.cc:407] Connection header received
0926 11:40:11.160069 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:11.160074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:11.160356 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:11.160553 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:11.161054 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.161899 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.162976 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.163219 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.163334 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:11.163734 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:11.164047 (+ 313us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:11.164155 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:11.164239 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 11:40:11.164305 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260926 11:40:11.165244 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:11.159330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.0.98.61:35059)
0926 11:40:11.159473 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:11.159479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:11.160225 (+ 746us) server_negotiation.cc:407] Connection header received
0926 11:40:11.160476 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:11.160481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:11.160538 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:11.160616 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:11.160993 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.161682 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.163437 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.163978 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.164089 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:11.164670 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:11.164778 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:11.164830 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:11.164999 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 11:40:11.165077 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:40:11.165452 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:11.159908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38388 (local address 127.0.98.60:46409)
0926 11:40:11.160044 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:11.160049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:11.160197 (+ 148us) server_negotiation.cc:407] Connection header received
0926 11:40:11.160324 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:11.160327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:11.160406 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:11.160489 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:11.160830 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.161647 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.163450 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:11.163978 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:11.164131 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:11.164668 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:11.164778 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:11.164846 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:11.165236 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 11:40:11.165319 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 11:40:11.169451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 22678
I20260926 11:40:11.178352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 22753
I20260926 11:40:11.186969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 22827
2026-09-26T11:40:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3238 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:40:11 dist-test-slave-4pw5 krb5kdc[23046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:40:11 dist-test-slave-4pw5 krb5kdc[23046](info): set up 2 sockets
Sep 26 11:40:11 dist-test-slave-4pw5 krb5kdc[23046](info): commencing operation
krb5kdc: starting...
W20260926 11:40:12.284811 392 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/test-user.keytab.
Sep 26 11:40:12 dist-test-slave-4pw5 krb5kdc[23046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:40:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:40:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:40:12.424018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34453
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34453,127.0.98.61:43917,127.0.98.60:35457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:40:12.553326 23059 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:40:12.553750 23059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:12.553808 23059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:12.553894 23059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:12.558710 23059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:12.558851 23059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:12.558890 23059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:12.558933 23059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:12.558980 23059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:12.565026 23059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34453,127.0.98.61:43917,127.0.98.60:35457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:12.566321 23059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:12.567399 23059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:12.577227 23064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:12.577262 23065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:12.577271 23067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:12.577574 23059 server_base.cc:1034] running on GCE node
I20260926 11:40:12.578312 23059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:12.579737 23059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:12.580968 23059 hybrid_clock.cc:648] HybridClock initialized: now 1790422812580945 us; error 45 us; skew 500 ppm
Sep 26 11:40:12 dist-test-slave-4pw5 krb5kdc[23046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:12.584398 23059 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:40:12.585580 23059 webserver.cc:460] Webserver started at http://127.0.98.62:42567/ using document root <none> and password file <none>
I20260926 11:40:12.586263 23059 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:12.586372 23059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:12.586649 23059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:12.589243 23059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/data/instance:
uuid: "a6b66a0645ca4cd4a67270876406810d"
format_stamp: "Formatted at 2026-09-26 11:40:12 on dist-test-slave-4pw5"
server_key: "0fb803fb9740d105c08844cba45f1e49"
server_key_iv: "4e4849b9841a74c260180862b4693c87"
server_key_version: "encryptionkey@0"
I20260926 11:40:12.589985 23059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6b66a0645ca4cd4a67270876406810d"
format_stamp: "Formatted at 2026-09-26 11:40:12 on dist-test-slave-4pw5"
server_key: "0fb803fb9740d105c08844cba45f1e49"
server_key_iv: "4e4849b9841a74c260180862b4693c87"
server_key_version: "encryptionkey@0"
I20260926 11:40:12.595382 23059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 11:40:12.598556 23074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:12.600101 23059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:40:12.600284 23059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/wal
uuid: "a6b66a0645ca4cd4a67270876406810d"
format_stamp: "Formatted at 2026-09-26 11:40:12 on dist-test-slave-4pw5"
server_key: "0fb803fb9740d105c08844cba45f1e49"
server_key_iv: "4e4849b9841a74c260180862b4693c87"
server_key_version: "encryptionkey@0"
I20260926 11:40:12.600432 23059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:12.613243 23059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:12.617364 23059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:12.617611 23059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:12.626045 23059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34453
I20260926 11:40:12.626482 23126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34453 every 8 connection(s)
I20260926 11:40:12.627182 23059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:40:12.630414 23127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:12.631946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 23059
I20260926 11:40:12.632148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-0/wal/instance
I20260926 11:40:12.632535 392 external_mini_cluster.cc:1428] Setting key 259229d1bd6afb2feaa26ee18e753463
I20260926 11:40:12.634619 23127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 11:40:12 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:12.654116 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.637707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34453 (local address 127.0.0.1:40494)
0926 11:40:12.638329 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.638393 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:12.639541 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:12.640687 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:12.640691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:12.641053 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:12.641342 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:12.641357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.642789 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.642793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:12.643819 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:12.643830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.644983 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.644991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:12.645124 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:12.645780 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:12.645798 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:12.647499 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:12.650267 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:12.650277 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:12.650286 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:12.650557 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:12.651045 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:12.651049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:12.651050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:12.651175 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:12.652121 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:12.652128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:12.652292 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 11:40:12.652643 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 11:40:12.653006 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":182,"threads_started":1}
I20260926 11:40:12.654120 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.638788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.0.98.62:34453)
0926 11:40:12.639202 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:12.639215 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:12.639269 (+ 54us) server_negotiation.cc:407] Connection header received
0926 11:40:12.640003 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:12.640010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:12.640325 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:12.640651 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:12.641518 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.642646 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.643986 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.644705 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.644945 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:12.647742 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:12.647748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:12.647759 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:12.647782 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:12.650087 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:12.650712 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:12.650722 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:12.650726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:12.650901 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:12.651298 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:12.651305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:12.651307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:12.651977 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:12.652137 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:12.652788 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 11:40:12.653198 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260926 11:40:12.657088 23127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:12.658092 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.657676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43917 (local address 127.0.0.1:50704)
0926 11:40:12.657855 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.658023 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 11:40:12.658504 23078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
I20260926 11:40:12.659704 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.659383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43917 (local address 127.0.0.1:50710)
0926 11:40:12.659505 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.659552 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 11:40:12.659921 23127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43917: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 11:40:12.663707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43917
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34453,127.0.98.61:43917,127.0.98.60:35457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:40:12.709200 23127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } attempt: 1
I20260926 11:40:12.710256 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.709911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43917 (local address 127.0.0.1:50718)
0926 11:40:12.710087 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.710163 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 11:40:12.711167 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.710932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43917 (local address 127.0.0.1:50722)
0926 11:40:12.711047 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.711101 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 11:40:12.711359 23127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43917: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
I20260926 11:40:12.779742 23127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } attempt: 2
W20260926 11:40:12.780104 23134 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:40:12.780462 23134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:12.780544 23134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:12.780628 23134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 11:40:12.780690 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.780334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43917 (local address 127.0.0.1:50736)
0926 11:40:12.780537 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.780609 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 11:40:12.781616 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.781304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43917 (local address 127.0.0.1:50750)
0926 11:40:12.781504 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.781556 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 11:40:12.781759 23127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43917: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43917: connect: Connection refused (error 111)
W20260926 11:40:12.785243 23134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:12.785357 23134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:12.785401 23134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:12.785442 23134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:12.785477 23134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:12.791225 23134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34453,127.0.98.61:43917,127.0.98.60:35457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:12.792502 23134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:12.793571 23134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:12.802659 23140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:12.802623 23142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:12.802623 23139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:12.802891 23134 server_base.cc:1034] running on GCE node
I20260926 11:40:12.803459 23134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:12.804890 23134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:12.806139 23134 hybrid_clock.cc:648] HybridClock initialized: now 1790422812806111 us; error 41 us; skew 500 ppm
Sep 26 11:40:12 dist-test-slave-4pw5 krb5kdc[23046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:12.809463 23134 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 11:40:12.810523 23134 webserver.cc:460] Webserver started at http://127.0.98.61:40259/ using document root <none> and password file <none>
I20260926 11:40:12.811118 23134 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:12.811182 23134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:12.811424 23134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:12.813618 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/data/instance:
uuid: "1ef662eb51ec42419847a163a7a432c5"
format_stamp: "Formatted at 2026-09-26 11:40:12 on dist-test-slave-4pw5"
server_key: "d685ee2e3d1e6b68ebf288433b14d718"
server_key_iv: "a00e4352a538c6f17ddf3be54203c553"
server_key_version: "encryptionkey@0"
I20260926 11:40:12.814201 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/wal/instance:
uuid: "1ef662eb51ec42419847a163a7a432c5"
format_stamp: "Formatted at 2026-09-26 11:40:12 on dist-test-slave-4pw5"
server_key: "d685ee2e3d1e6b68ebf288433b14d718"
server_key_iv: "a00e4352a538c6f17ddf3be54203c553"
server_key_version: "encryptionkey@0"
I20260926 11:40:12.818132 23134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:40:12.820991 23149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:12.822953 23134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:40:12.823103 23134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/wal
uuid: "1ef662eb51ec42419847a163a7a432c5"
format_stamp: "Formatted at 2026-09-26 11:40:12 on dist-test-slave-4pw5"
server_key: "d685ee2e3d1e6b68ebf288433b14d718"
server_key_iv: "a00e4352a538c6f17ddf3be54203c553"
server_key_version: "encryptionkey@0"
I20260926 11:40:12.823290 23134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:12.839701 23134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:12.843528 23134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:12.843884 23134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:12.851824 23134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43917
I20260926 11:40:12.852169 23201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43917 every 8 connection(s)
I20260926 11:40:12.853288 23134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:40:12.856660 23202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:12.860800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 23134
I20260926 11:40:12.860668 23202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:12.861095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-1/wal/instance
I20260926 11:40:12.861449 392 external_mini_cluster.cc:1428] Setting key fcafc40417344142c1d8a269113efd32
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 11:40:12 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:12.872929 23127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } attempt: 3
I20260926 11:40:12.878703 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.862847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.0.98.62:34453)
0926 11:40:12.863078 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:12.863082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:12.864182 (+ 1100us) server_negotiation.cc:407] Connection header received
0926 11:40:12.865267 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:12.865271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:12.865339 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:12.865450 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:12.866767 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.867331 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.868612 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.868867 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.869002 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:12.874075 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:12.874078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:12.874080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:12.874104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:12.876051 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:12.876777 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:12.876781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:12.876783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:12.876887 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:12.877360 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:12.877381 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:12.877386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:12.877663 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:12.877775 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:12.878394 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 11:40:12.878523 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 11:40:12.879814 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.863368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34453 (local address 127.0.0.1:40510)
0926 11:40:12.863998 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.864047 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:12.865033 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:12.865679 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:12.865686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:12.866092 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:12.866602 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:12.866617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.867491 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.867494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:12.868467 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:12.868478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.869050 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.869054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:12.869281 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:12.870023 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:12.870041 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:12.873552 (+ 3511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:12.876174 (+ 2622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:12.876180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:12.876193 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:12.876641 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:12.877018 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:12.877022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:12.877024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:12.877227 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:12.877821 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:12.877830 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:12.878124 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 11:40:12.878323 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 11:40:12.878682 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":196,"threads_started":1}
Sep 26 11:40:12 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:40:12.882187 23202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:12.886230 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.873538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43917 (local address 127.0.0.1:50764)
0926 11:40:12.873685 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.873702 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:12.873805 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:12.874622 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:12.874625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:12.874974 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:12.875400 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:12.875414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.876985 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.876991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:12.877738 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:12.877750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.878678 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.878686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:12.878833 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:12.879363 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:12.879382 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:12.881540 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:12.884332 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:12.884339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:12.884344 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:12.884655 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:12.884981 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:12.884983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:12.884985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:12.885076 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:12.885638 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:12.885641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:12.885749 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 11:40:12.885857 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 11:40:12.886002 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 11:40:12.886718 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.873661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.0.98.61:43917)
0926 11:40:12.874042 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:12.874057 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:12.874090 (+ 33us) server_negotiation.cc:407] Connection header received
0926 11:40:12.874148 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:12.874158 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:12.874414 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:12.874840 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:12.875546 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.876845 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.877860 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.878512 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.879119 (+ 607us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:12.881739 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:12.881748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:12.881762 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:12.881791 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:12.884207 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:12.884775 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:12.884782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:12.884783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:12.884873 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:12.885197 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:12.885203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:12.885208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:12.885525 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:12.885631 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:12.886292 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 11:40:12.886449 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
Sep 26 11:40:12 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:40:12.889020 23127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:12.890012 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.889655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44404)
0926 11:40:12.889812 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.889886 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 11:40:12.890830 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.890556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44418)
0926 11:40:12.890726 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.890778 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 11:40:12.891021 23127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
I20260926 11:40:12.893334 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.882847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43917 (local address 127.0.0.1:50768)
0926 11:40:12.883432 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.883447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:12.883536 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:12.883821 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:12.883824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:12.884148 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:12.884247 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:12.884279 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.885292 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.885295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:12.886006 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:12.886016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.886698 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.886701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:12.886812 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:12.887268 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:12.887281 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:12.889399 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:12.891683 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:12.891689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:12.891691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:12.891926 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:12.892207 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:12.892210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:12.892212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:12.892291 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:12.892716 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:12.892719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:12.892802 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 11:40:12.892919 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 11:40:12.893112 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507}
I20260926 11:40:12.893388 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.882847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.0.98.61:43917)
0926 11:40:12.883466 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:12.883478 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:12.883538 (+ 60us) server_negotiation.cc:407] Connection header received
0926 11:40:12.883663 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:12.883666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:12.883713 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:12.883812 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:12.884409 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.885143 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.886145 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:12.886489 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:12.886593 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:12.889582 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:12.889585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:12.889587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:12.889608 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:12.891537 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:12.892011 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:12.892014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:12.892016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:12.892104 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:12.892385 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:12.892388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:12.892390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:12.892622 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:12.892731 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:12.892962 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 11:40:12.893179 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":561,"thread_start_us":429,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 11:40:12.894276 23202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:12.895033 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.894755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44434)
0926 11:40:12.894875 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.894966 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 11:40:12.895076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
/tmp/dist-test-taskPt_vxu/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35457
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34453,127.0.98.61:43917,127.0.98.60:35457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:40:12.895260 23151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
I20260926 11:40:12.896288 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.895978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44448)
0926 11:40:12.896121 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.896194 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 11:40:12.896709 23202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
I20260926 11:40:12.922233 23127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } attempt: 1
I20260926 11:40:12.923337 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.922984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44454)
0926 11:40:12.923153 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.923234 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 11:40:12.924254 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.924001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44458)
0926 11:40:12.924134 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.924198 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 11:40:12.924439 23127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
I20260926 11:40:12.946244 23202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } attempt: 1
I20260926 11:40:12.947386 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.946999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44470)
0926 11:40:12.947194 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.947282 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 11:40:12.948302 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:12.948032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44482)
0926 11:40:12.948168 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:12.948228 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 11:40:12.948596 23202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
I20260926 11:40:12.999670 23127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } attempt: 2
I20260926 11:40:13.000620 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.000276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44488)
0926 11:40:13.000445 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:13.000524 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 11:40:13.001628 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.001336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44494)
0926 11:40:13.001509 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:13.001570 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 11:40:13.001816 23127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
I20260926 11:40:13.016790 23202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } attempt: 2
I20260926 11:40:13.017845 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.017445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44504)
0926 11:40:13.017639 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:13.017717 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 11:40:13.018680 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.018451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44516)
0926 11:40:13.018576 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:13.018625 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 11:40:13.018846 23202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35457: connect: Connection refused (error 111)
W20260926 11:40:13.019111 23210 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:40:13.019407 23210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:40:13.019439 23210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:40:13.019546 23210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:40:13.024649 23210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:40:13.024729 23210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:40:13.024752 23210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:40:13.024773 23210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:40:13.024838 23210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:40:13.031739 23210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34453,127.0.98.61:43917,127.0.98.60:35457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:40:13.032974 23210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:40:13.034090 23210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:40:13.043133 23215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:40:13.043136 23216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:13.043264 23210 server_base.cc:1034] running on GCE node
W20260926 11:40:13.043133 23218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:40:13.044176 23210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:40:13.045380 23210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:40:13.046593 23210 hybrid_clock.cc:648] HybridClock initialized: now 1790422813046553 us; error 59 us; skew 500 ppm
Sep 26 11:40:13 dist-test-slave-4pw5 krb5kdc[23046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:40:13.050138 23210 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 11:40:13.051412 23210 webserver.cc:460] Webserver started at http://127.0.98.60:40607/ using document root <none> and password file <none>
I20260926 11:40:13.052140 23210 fs_manager.cc:359] Metadata directory not provided
I20260926 11:40:13.052224 23210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:40:13.052505 23210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:40:13.055157 23210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/data/instance:
uuid: "ea8fef496a184e83b65943944611388d"
format_stamp: "Formatted at 2026-09-26 11:40:13 on dist-test-slave-4pw5"
server_key: "4b2bf21cfff224342f5f5207b99f0c78"
server_key_iv: "5a705e93d7d31584e5a9103eb9061142"
server_key_version: "encryptionkey@0"
I20260926 11:40:13.055891 23210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/wal/instance:
uuid: "ea8fef496a184e83b65943944611388d"
format_stamp: "Formatted at 2026-09-26 11:40:13 on dist-test-slave-4pw5"
server_key: "4b2bf21cfff224342f5f5207b99f0c78"
server_key_iv: "5a705e93d7d31584e5a9103eb9061142"
server_key_version: "encryptionkey@0"
I20260926 11:40:13.060381 23210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 11:40:13.063432 23225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:40:13.064920 23210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:40:13.065094 23210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/wal
uuid: "ea8fef496a184e83b65943944611388d"
format_stamp: "Formatted at 2026-09-26 11:40:13 on dist-test-slave-4pw5"
server_key: "4b2bf21cfff224342f5f5207b99f0c78"
server_key_iv: "5a705e93d7d31584e5a9103eb9061142"
server_key_version: "encryptionkey@0"
I20260926 11:40:13.065232 23210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:40:13.086758 23210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:40:13.090509 23210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:40:13.090747 23210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:40:13.098896 23210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35457
I20260926 11:40:13.099236 23277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35457 every 8 connection(s)
I20260926 11:40:13.100028 23210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/data/info.pb
I20260926 11:40:13.101043 23127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } attempt: 3
I20260926 11:40:13.103267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu as pid 23210
I20260926 11:40:13.103394 23278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:40:13.103426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPt_vxu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422511662684-392-0/minicluster-data/master-2/wal/instance
I20260926 11:40:13.103803 392 external_mini_cluster.cc:1428] Setting key 6101d836d5d80e1e0575782d93b52652
I20260926 11:40:13.109436 23278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:13.110052 23202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } attempt: 3
Sep 26 11:40:13 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:40:13 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:40:13.116861 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.101624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44522)
0926 11:40:13.101761 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:13.101777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:13.101910 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:13.104855 (+ 2945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:13.104861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:13.105145 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:13.105271 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.105279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.107288 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.107291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.108081 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.108095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.109236 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.109240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.109328 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.109901 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:13.109924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:13.111556 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:13.114147 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.114150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.114152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.114425 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.114795 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.114798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.114800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.114905 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.116075 (+ 1170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:13.116078 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:13.116183 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 11:40:13.116289 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 11:40:13.116439 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
Sep 26 11:40:13 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:40:13.118412 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.102382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.0.98.60:35457)
0926 11:40:13.102820 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.102839 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.102882 (+ 43us) server_negotiation.cc:407] Connection header received
0926 11:40:13.103830 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.103853 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.104291 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.104902 (+ 611us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.105381 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.107100 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.108249 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.109067 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.109256 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.111817 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.111824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.111854 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.111887 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.114021 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.114548 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.114554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.114560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.114695 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.115003 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.115006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.115008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.115561 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.115725 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.116739 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.117100 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260926 11:40:13.121870 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.110558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44542)
0926 11:40:13.110861 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:13.110877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:13.110964 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:13.111791 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:13.111795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:13.112003 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:13.112669 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.112683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.114024 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.114028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.114862 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.114872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.115485 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.115490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.115641 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.116066 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:13.116090 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:13.117719 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:13.120252 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.120256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.120258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.120519 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.120788 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.120791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.120793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.120898 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.121337 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:13.121341 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:13.121454 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 11:40:13.121544 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 11:40:13.121685 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260926 11:40:13.121981 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.104561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.0.98.60:35457)
0926 11:40:13.107471 (+ 2910us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.107482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.107499 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:40:13.107541 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.107544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.107644 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.107755 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.108733 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.109399 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.111080 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.111502 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.111724 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.115333 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.115337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.115339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.115364 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.119298 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.119935 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.119940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.119944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.120039 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.120484 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.120488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.120490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.120758 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.120941 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.121555 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.121717 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2816,"thread_start_us":143,"threads_started":1}
Sep 26 11:40:13 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:13.122884 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.111121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.0.98.60:35457)
0926 11:40:13.111505 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.111514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.111528 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:40:13.111593 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.111596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.111644 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.112439 (+ 795us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.113050 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.113859 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.115003 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.115337 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.115441 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.117901 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.117905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.117908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.117935 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.120131 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.120621 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.120623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.120624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.120689 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.121007 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.121011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.121012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.121219 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.122447 (+ 1228us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.122544 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.122673 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":137,"threads_started":1}
I20260926 11:40:13.125012 23127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d: Bootstrap starting.
I20260926 11:40:13.127341 23202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5: Bootstrap starting.
I20260926 11:40:13.127796 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.112130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34453 (local address 127.0.0.1:40524)
0926 11:40:13.113879 (+ 1749us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:13.113957 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:13.114173 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:13.114690 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:13.114697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:13.115203 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:13.115940 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.115949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.116952 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.116955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.117968 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.117977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.118764 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.118769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.118921 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.119998 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:13.121033 (+ 1035us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:13.123188 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:13.125659 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.125672 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.125678 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.126035 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.126424 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.126428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.126430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.126522 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.127052 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:13.127059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:13.127189 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 11:40:13.127351 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 11:40:13.127612 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":475,"thread_start_us":147,"threads_started":1}
I20260926 11:40:13.128496 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.111828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.0.98.62:34453)
0926 11:40:13.111968 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.111974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.114053 (+ 2079us) server_negotiation.cc:407] Connection header received
0926 11:40:13.114342 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.114346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.114405 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.114471 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.116100 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.116808 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.118103 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.118493 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.118584 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.123417 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.123423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.123426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.123463 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.125521 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.126180 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.126184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.126185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.126298 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.126627 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.126630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.126632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.126899 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.126995 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.128204 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.128345 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:40:13.129284 23127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:13.129594 23278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:13.130388 23202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:13.130726 23127 log.cc:824] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d: Log is configured to *not* fsync() on all Append() calls
I20260926 11:40:13.131306 23202 log.cc:824] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:40:13 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:40:13.133517 23127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d: No bootstrap required, opened a new log
I20260926 11:40:13.134141 23202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5: No bootstrap required, opened a new log
Sep 26 11:40:13 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:40:13.137661 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.123640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.0.98.62:34453)
0926 11:40:13.124297 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.124324 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.124342 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:40:13.124387 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.124391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.124445 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.124552 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.125612 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.126582 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.127959 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.128454 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.128591 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.132821 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.132826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.132830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.132862 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.134930 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.135563 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.135566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.135605 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.135701 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.136030 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.136033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.136035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.136264 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.136473 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.137197 (+ 724us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.137431 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":91,"threads_started":1}
I20260926 11:40:13.138993 23127 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } }
I20260926 11:40:13.139236 23202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } }
I20260926 11:40:13.139467 23127 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b66a0645ca4cd4a67270876406810d, State: Initialized, Role: FOLLOWER
I20260926 11:40:13.139568 23202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef662eb51ec42419847a163a7a432c5, State: Initialized, Role: FOLLOWER
I20260926 11:40:13.139983 23127 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } }
I20260926 11:40:13.140025 23202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } }
I20260926 11:40:13.141213 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.130089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43917 (local address 127.0.0.1:50772)
0926 11:40:13.130348 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:13.130363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:13.130453 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:13.130838 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:13.130841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:13.131029 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:13.131153 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.131162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.132227 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.132234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.133209 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.133223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.134063 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.134067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.134199 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.134722 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:13.134742 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:13.136573 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:13.139358 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.139362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.139364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.139698 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.140040 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.140046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.140048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.140141 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.140679 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:13.140682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:13.140781 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 11:40:13.140886 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 11:40:13.141043 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260926 11:40:13.141821 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.130208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.0.98.61:43917)
0926 11:40:13.130359 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.130367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.130538 (+ 171us) server_negotiation.cc:407] Connection header received
0926 11:40:13.130584 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.130587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.130639 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.130718 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.131293 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.132091 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.133370 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.133766 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.133906 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.136792 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.136797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.136800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.136837 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.139181 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.139832 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.139836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.139838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.139932 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.140252 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.140256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.140257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.140539 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.140635 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.141509 (+ 874us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.141637 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:40:13.141697 23202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:13.142294 23127 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:13.142766 23278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:40:13.146260 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.146661 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 11:40:13 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260926 11:40:13.150642 23310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:40:13.150797 23310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:40:13.151058 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.151355 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 11:40:13.151674 23311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:40:13.151799 23311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:40:13.153627 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.143357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35457 (local address 127.0.0.1:44558)
0926 11:40:13.143752 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 11:40:13.143769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:40:13.143871 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:40:13.144212 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:40:13.144215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:40:13.144426 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:40:13.144543 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.144552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.145579 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.145582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.146310 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:40:13.146321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.147002 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.147005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:40:13.147079 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.148092 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:40:13.148109 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:40:13.149940 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:40:13.152014 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.152016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.152019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.152254 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.152559 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:40:13.152562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:40:13.152564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:40:13.152645 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:40:13.153082 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:40:13.153088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:40:13.153181 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 11:40:13.153284 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 11:40:13.153443 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260926 11:40:13.154170 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.143612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.0.98.60:35457)
0926 11:40:13.143748 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.143754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.143822 (+ 68us) server_negotiation.cc:407] Connection header received
0926 11:40:13.143980 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.143983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.144024 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.144074 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.144673 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.145457 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.146447 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.146765 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.146846 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.150115 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.150119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.150122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.150152 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.151898 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.152376 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.152379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.152381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.152467 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.152730 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.152733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.152734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.152949 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.153047 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.153898 (+ 851us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.154027 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 11:40:13.157262 23278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d: Bootstrap starting.
I20260926 11:40:13.160241 23278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d: Neither blocks nor log segments found. Creating new log.
I20260926 11:40:13.156915 23059 master_runner.cc:186] Error getting master registration for 127.0.98.60:35457: OK, instance_id { permanent_uuid: "ea8fef496a184e83b65943944611388d" instance_seqno: 1790422813096575 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:40:13.160434 23059 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:40:13.161218 23278 log.cc:824] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:40:13 dist-test-slave-4pw5 krb5kdc[23046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:40:13.154736 23134 master_runner.cc:186] Error getting master registration for 127.0.98.60:35457: OK, instance_id { permanent_uuid: "ea8fef496a184e83b65943944611388d" instance_seqno: 1790422813096575 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:40:13.164964 23134 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:40:13.167950 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.140368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.0.98.61:43917)
0926 11:40:13.140511 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.140518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.140534 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:40:13.140701 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.140706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.140771 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.140837 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.141930 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.142671 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.148835 (+ 6164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.149181 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.149263 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.164237 (+ 14974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.164243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.164245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.164286 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.166113 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.166618 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.166621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.166623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.166712 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.167007 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.167010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.167012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.167218 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.167462 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.167625 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.167766 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:40:13.171543 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:40:13.173105 23101 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a6b66a0645ca4cd4a67270876406810d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40528
I20260926 11:40:13.173398 23101 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 11:40:13.173602 23101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:13.174338 23278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d: No bootstrap required, opened a new log
I20260926 11:40:13.174988 23101 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } }
I20260926 11:40:13.177202 23174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a6b66a0645ca4cd4a67270876406810d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ef662eb51ec42419847a163a7a432c5"
I20260926 11:40:13.177515 23174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:13.179095 23174 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b66a0645ca4cd4a67270876406810d in term 1.
I20260926 11:40:13.179659 23278 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } }
I20260926 11:40:13.179841 23078 leader_election.cc:302] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ef662eb51ec42419847a163a7a432c5, a6b66a0645ca4cd4a67270876406810d; no voters:
I20260926 11:40:13.179960 23278 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8fef496a184e83b65943944611388d, State: Initialized, Role: FOLLOWER
I20260926 11:40:13.180176 23289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:40:13.180316 23278 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } }
I20260926 11:40:13.180439 23289 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [term 1 LEADER]: Becoming Leader. State: Replica: a6b66a0645ca4cd4a67270876406810d, State: Running, Role: LEADER
I20260926 11:40:13.180518 23101 leader_election.cc:288] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [CANDIDATE]: Term 1 election: Requested vote from peers 1ef662eb51ec42419847a163a7a432c5 (127.0.98.61:43917), ea8fef496a184e83b65943944611388d (127.0.98.60:35457)
I20260926 11:40:13.180787 23289 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } }
I20260926 11:40:13.181834 23278 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:40:13.182650 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6b66a0645ca4cd4a67270876406810d. Latest consensus state: current_term: 1 leader_uuid: "a6b66a0645ca4cd4a67270876406810d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.182763 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:13.185550 23252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a6b66a0645ca4cd4a67270876406810d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ea8fef496a184e83b65943944611388d"
I20260926 11:40:13.185855 23252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:40:13.187247 23252 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b66a0645ca4cd4a67270876406810d in term 1.
I20260926 11:40:13.193657 23321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:40:13.194835 23321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:40:13.197978 23316 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.198328 23316 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:13.205766 23252 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [term 1 FOLLOWER]: Refusing update from remote peer a6b66a0645ca4cd4a67270876406810d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:40:13.206662 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 11:40:13.208258 23329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 11:40:13.212790 23174 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [term 1 FOLLOWER]: Refusing update from remote peer a6b66a0645ca4cd4a67270876406810d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:40:13.213589 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:40:13.211324 23316 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6b66a0645ca4cd4a67270876406810d. Latest consensus state: current_term: 1 leader_uuid: "a6b66a0645ca4cd4a67270876406810d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.216024 23316 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 11:40:13.216179 23329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:40:13.219159 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6b66a0645ca4cd4a67270876406810d. Latest consensus state: current_term: 1 leader_uuid: "a6b66a0645ca4cd4a67270876406810d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.219336 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:13.222039 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.189601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.0.98.62:34453)
0926 11:40:13.190137 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.190143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.191080 (+ 937us) server_negotiation.cc:407] Connection header received
0926 11:40:13.191281 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.191287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.191341 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.191395 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.202867 (+ 11472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.203635 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.208788 (+ 5153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.209152 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.209252 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.216881 (+ 7629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.216886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.216890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.216929 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.219929 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.220557 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.220561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.220563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.220661 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.220977 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.220981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.220983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.221233 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.221369 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.221720 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.221866 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450}
I20260926 11:40:13.225289 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.204052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.0.98.60:35457)
0926 11:40:13.204213 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.204217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.208645 (+ 4428us) server_negotiation.cc:407] Connection header received
0926 11:40:13.208832 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.208836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.208894 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.208950 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.209868 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.210627 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.215232 (+ 4605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.215685 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.215776 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.220674 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.220678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.220682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.220712 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.222658 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.223614 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.223622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.223626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.223730 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.224144 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.224149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.224151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.224425 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.224601 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.224973 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.225106 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 11:40:13.225946 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:13.190455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.0.98.61:43917)
0926 11:40:13.190577 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.190581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.207890 (+ 17309us) server_negotiation.cc:407] Connection header received
0926 11:40:13.208105 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.208109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.208164 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.208216 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.215680 (+ 7464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.216549 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.217746 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:13.218081 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:13.218165 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:13.221570 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:40:13.221827 (+ 257us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:40:13.221832 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:40:13.221870 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:40:13.223884 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.224451 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.224456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.224458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.224553 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:40:13.224871 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:40:13.224875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:40:13.224877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:40:13.225105 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:40:13.225229 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:13.225629 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 11:40:13.225772 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:40:13.229912 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6b66a0645ca4cd4a67270876406810d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.230065 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:13.231623 23316 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6b66a0645ca4cd4a67270876406810d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.231766 23316 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:13.232450 23336 mvcc.cc:205] Tried to move back new op lower bound from 7333571842880163840 to 7333571842792050688. Current Snapshot: MvccSnapshot[applied={T|T < 7333571842880163840}]
I20260926 11:40:13.232661 23339 mvcc.cc:205] Tried to move back new op lower bound from 7333571842880163840 to 7333571842792050688. Current Snapshot: MvccSnapshot[applied={T|T < 7333571842880163840}]
I20260926 11:40:13.236542 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6b66a0645ca4cd4a67270876406810d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.236716 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d [sys.catalog]: This master's current role is: LEADER
I20260926 11:40:13.238300 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6b66a0645ca4cd4a67270876406810d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b66a0645ca4cd4a67270876406810d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34453 } } peers { permanent_uuid: "1ef662eb51ec42419847a163a7a432c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43917 } } peers { permanent_uuid: "ea8fef496a184e83b65943944611388d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35457 } } }
I20260926 11:40:13.238443 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:40:13.239071 23341 mvcc.cc:205] Tried to move back new op lower bound from 7333571842880163840 to 7333571842792050688. Current Snapshot: MvccSnapshot[applied={T|T < 7333571842880163840}]
I20260926 11:40:13.245713 23321 catalog_manager.cc:1347] Generated new cluster ID: 8332f3aafa6a4d98b3509849851f3727
I20260926 11:40:13.245812 23321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:40:13.266820 23321 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:40:13.270843 23321 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:40:13.295161 23321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6b66a0645ca4cd4a67270876406810d: Generated new TSK 0
I20260926 11:40:13.296402 23321 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:40:13.329798 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.326291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.0.98.61:43917)
0926 11:40:13.326438 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.326443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.328489 (+ 2046us) server_negotiation.cc:407] Connection header received
0926 11:40:13.328728 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.328732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.328791 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.328857 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.329626 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50796: BlockingRecv error: recv got EOF from 127.0.0.1:50796 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 11:40:13.332154 23285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.326960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.0.98.62:34453)
0926 11:40:13.327090 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.327095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.330454 (+ 3359us) server_negotiation.cc:407] Connection header received
0926 11:40:13.331150 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.331154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.331216 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.331275 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.332008 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40546: BlockingRecv error: recv got EOF from 127.0.0.1:40546 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 11:40:13.333628 23283 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.327818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.0.98.60:35457)
0926 11:40:13.327958 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.327963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.330980 (+ 3017us) server_negotiation.cc:407] Connection header received
0926 11:40:13.331089 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.331093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.331153 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.331257 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.333487 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44570: BlockingRecv error: recv got EOF from 127.0.0.1:44570 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 11:40:13.339361 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.337516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.0.98.61:43917)
0926 11:40:13.337646 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.337651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.338162 (+ 511us) server_negotiation.cc:407] Connection header received
0926 11:40:13.338438 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.338442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.338541 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.338607 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.339213 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50802: BlockingRecv error: recv got EOF from 127.0.0.1:50802 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 11:40:13.346943 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.344895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.0.98.61:43917)
0926 11:40:13.345072 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.345078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.345776 (+ 698us) server_negotiation.cc:407] Connection header received
0926 11:40:13.346012 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.346016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.346101 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.346164 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.346812 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50804: BlockingRecv error: recv got EOF from 127.0.0.1:50804 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260926 11:40:13.353646 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.351642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.0.98.61:43917)
0926 11:40:13.351778 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.351784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.352432 (+ 648us) server_negotiation.cc:407] Connection header received
0926 11:40:13.352664 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.352668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.352731 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.352792 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.353508 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50818: BlockingRecv error: recv got EOF from 127.0.0.1:50818 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 11:40:13.362462 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.360474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.0.98.61:43917)
0926 11:40:13.360651 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.360656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.361303 (+ 647us) server_negotiation.cc:407] Connection header received
0926 11:40:13.361558 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.361562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.361624 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.361684 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.362306 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50832: BlockingRecv error: recv got EOF from 127.0.0.1:50832 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260926 11:40:13.375363 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.373376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.0.98.61:43917)
0926 11:40:13.373525 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.373531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.374193 (+ 662us) server_negotiation.cc:407] Connection header received
0926 11:40:13.374441 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.374445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.374505 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.374567 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.375220 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50840: BlockingRecv error: recv got EOF from 127.0.0.1:50840 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 11:40:13.396559 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.394371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.0.98.61:43917)
0926 11:40:13.394523 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.394530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.395270 (+ 740us) server_negotiation.cc:407] Connection header received
0926 11:40:13.395554 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.395558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.395671 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.395734 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.396410 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50854: BlockingRecv error: recv got EOF from 127.0.0.1:50854 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 11:40:13.432843 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.431357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.0.98.61:43917)
0926 11:40:13.431494 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.431499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.431779 (+ 280us) server_negotiation.cc:407] Connection header received
0926 11:40:13.431922 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.431929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.431990 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.432071 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.432685 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50858: BlockingRecv error: recv got EOF from 127.0.0.1:50858 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 11:40:13.500604 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.499095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.0.98.61:43917)
0926 11:40:13.499264 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.499268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.499369 (+ 101us) server_negotiation.cc:407] Connection header received
0926 11:40:13.499702 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.499712 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.499768 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.499852 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.500494 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50860: BlockingRecv error: recv got EOF from 127.0.0.1:50860 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 11:40:13.632223 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.630891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.0.98.61:43917)
0926 11:40:13.631054 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.631061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.631280 (+ 219us) server_negotiation.cc:407] Connection header received
0926 11:40:13.631377 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.631380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.631477 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.631556 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.632097 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50870: BlockingRecv error: recv got EOF from 127.0.0.1:50870 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 11:40:13.892940 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:40:13.891394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.0.98.61:43917)
0926 11:40:13.891539 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:13.891543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:13.891664 (+ 121us) server_negotiation.cc:407] Connection header received
0926 11:40:13.891842 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:13.891845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:13.891904 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:13.891999 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:40:13.892766 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50876: BlockingRecv error: recv got EOF from 127.0.0.1:50876 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:40:14.153283 23311 catalog_manager.cc:1259] Loaded cluster ID: 8332f3aafa6a4d98b3509849851f3727
I20260926 11:40:14.153383 23311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5: loading cluster ID for follower catalog manager: success
I20260926 11:40:14.155282 23311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5: acquiring CA information for follower catalog manager: success
I20260926 11:40:14.156561 23311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ef662eb51ec42419847a163a7a432c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:40:14.218432 23329 catalog_manager.cc:1259] Loaded cluster ID: 8332f3aafa6a4d98b3509849851f3727
I20260926 11:40:14.218549 23329 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d: loading cluster ID for follower catalog manager: success
I20260926 11:40:14.220551 23329 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d: acquiring CA information for follower catalog manager: success
I20260926 11:40:14.221860 23329 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea8fef496a184e83b65943944611388d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:40:14.411966 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:14.407477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.0.98.61:43917)
0926 11:40:14.407845 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:14.407848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:14.407872 (+ 24us) server_negotiation.cc:407] Connection header received
0926 11:40:14.408079 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:14.408082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:14.408218 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:14.408355 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:14.408728 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.409300 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.410351 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.410588 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.410682 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:14.411032 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:14.411479 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:14.411645 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:14.411775 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 11:40:14.411836 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":200,"threads_started":1}
I20260926 11:40:14.420284 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:14.416010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.0.98.60:35457)
0926 11:40:14.416270 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:14.416274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:14.416409 (+ 135us) server_negotiation.cc:407] Connection header received
0926 11:40:14.416556 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:14.416558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:14.416710 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:14.416827 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:14.417135 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.417695 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.418696 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.418915 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.418998 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:14.419354 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:14.419889 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:14.419991 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:14.420097 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 11:40:14.420151 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":110,"threads_started":1}
I20260926 11:40:14.428512 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:14.423760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.0.98.61:43917)
0926 11:40:14.423905 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:14.423909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:14.424371 (+ 462us) server_negotiation.cc:407] Connection header received
0926 11:40:14.424467 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:14.424471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:14.424589 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:14.424690 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:14.425019 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.425765 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.426896 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.427297 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.427411 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:14.427947 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:14.428062 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:14.428147 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:14.428249 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 11:40:14.428325 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:40:14.428747 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:14.424163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.0.98.60:35457)
0926 11:40:14.424323 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:14.424328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:14.424606 (+ 278us) server_negotiation.cc:407] Connection header received
0926 11:40:14.424772 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:14.424776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:14.424842 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:14.424896 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:14.425313 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.426246 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.427365 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.427720 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.427833 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:14.428143 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:14.428237 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:14.428321 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:14.428518 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 11:40:14.428603 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 11:40:14.429701 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:40:14.423825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.0.98.62:34453)
0926 11:40:14.424238 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 11:40:14.424243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:40:14.424691 (+ 448us) server_negotiation.cc:407] Connection header received
0926 11:40:14.425011 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:40:14.425015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:40:14.425187 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:40:14.425354 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:40:14.425562 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.426438 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.427773 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:40:14.428189 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:40:14.428311 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:40:14.428887 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:40:14.429234 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:40:14.429345 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:40:14.429425 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 11:40:14.429494 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":168,"threads_started":1}
I20260926 11:40:14.439265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 23059
I20260926 11:40:14.446790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 23134
I20260926 11:40:14.454335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPt_vxu/build/debug/bin/kudu with pid 23210
2026-09-26T11:40:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3266 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6505 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (302780 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST